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A  COMPARISON  OF  INTERACTIVE  MULTIMEDIA  INSTRUCTION  DESIGNS 
ADDRESSING  SOLDIERS’  LEARNING  NEEDS 

EXECUTIVE  SUMMARY 


Research  Requirement: 

This  is  the  third  in  a  series  of  three  reports  on  a  research  effort  that  (a)  evaluated  existing 
Army  Combat  Arms  interactive  multimedia  instruction  (IMI)  for  modification  to  address 
learners’  needs  (i.e.,  Blankenbeckler,  Graves,  &  Wampler,  2013),  (b)  designed  and  developed  six 
IMI  exemplars  to  cover  three  variations  of  needs-focused  IMI  designs  (i.e.,  familiarization, 
core/refresher,  and  tailored  training)  for  two  topics  (i.e.,  Adjust  Indirect  Fire  and  Conduct  a 
Defense  by  a  Squad)  (i.e.,  Blankenbeckler,  Graves,  &  Wampler,  2014),  and  (c)  experimentally 
evaluated  the  IMI  exemplars  with  Soldiers’  enrolled  in  the  Warrior  Leaders  Course  at  Fort 
Benning,  GA.  This  report  is  focused  on  the  experimental  evaluation  of  the  IMI  exemplars. 
Together,  the  three  reports  and  the  IMI  exemplars  address  challenges  the  Army  faces  to  improve 
warfighting  capabilities  through  enhanced  training  using  tailored  distributed  training  and 
education,  intended  to  prepare  and  support  Soldiers’  and  Leaders’  learning  at  their  point  of  need 
(TRADOC,  2014). 

IMI  enables  the  Army  to  provide  training  to  Soldiers  when  and  where  they  need  it 
(TRADOC,  2011),  with  content  and  instructional  design  to  address  Soldiers’  learning  needs.  In 
addition,  IMI  provides  a  viable  means  to  deliver  training  that  is  tailored  to  individuals’  specific 
learning  needs.  Often,  implementations  of  tailored  or  adaptive  training  in  IMI  are  complex, 
requiring  detailed  statistical  analyses,  or  are  too  costly  to  justify  development  for  a  small, 
narrowly-defined  audience.  However,  there  may  be  simpler  yet  equally  effective  ways  to  design 
IMI  to  address  the  learning  needs  of  both  a  narrowly  defined  audience  and  individual  learners, 
targeting  what  has  been  described  in  the  Army  Learning  Model  (ALM)  as  point  of  need.  This 
research  focused  on  experimentally  testing  three  types  of  needs-focused  IMI  designs  in  order  to 
determine  which  may  be  the  most  effective  in  addressing  Soldiers’  learning  needs. 

Procedure: 

Six  IMI  exemplars  were  developed,  focusing  on  three  types  of  learners’  needs,  i.e., 
familiarization,  core/refresher,  and  tailored  training.  Familiarization  IMI  focused  content  on 
breadth  of  information  to  help  learners  become  familiar  with  a  topic,  and  prepare  them  for 
additional  learning.  Core/refresher  IMI  content  focused  less  on  breadth  and  more  on  depth  of 
information  to  support  learners  in  executing  a  specific  task.  Finally,  tailored  training 
incorporated  aspects  of  both  familiarization  and  core/refresher  in  terms  of  content,  including  pre- 
and  post-training  diagnostic  assessments  with  individualized  feedback.  The  diagnostic 
assessment  and  individualized  feedback  was  intended  to  assist  Soldiers  in  selecting  the  most 
beneficial  learning  path  for  them  among  the  various  options  made  available.  The  familiarization 
and  core/refresher  IMI  were  non-tailored.  Targeting  new  squad/team  leaders  (i.e.,  E-4 
Specialist/Corporal  and  E-5  Sergeant),  the  two  training  topics  selected  were  Adjust  Indirect  Fire 
(a  topic  less  familiar  to  the  intended  audience),  and  Conduct  a  Defense  by  a  Squad  (a  topic  more 
familiar  to  the  intended  audience)  (Blankenbeckler  et  al.,  2014). 


IV 


The  IMI  were  tested  in  six  sessions  with  91  Soldiers  enrolled  in  the  Warrior  Leader 
Course  at  Fort  Benning,  GA.  Soldiers  were  administered  a  self-report  questionnaire  intended  to 
estimate  their  background  knowledge  and  experience  with  the  respective  topics.  Their  responses 
to  the  questionnaire  were  used  to  assign  them  to  the  three  IMI  conditions,  balancing  the  level  of 
knowledge  and  experience  within  each  group.  Soldiers  were  then  administered  a  pretest  to 
measure  their  level  of  knowledge  before  training.  Following  the  pretest,  they  completed  the  IMI 
training.  After  IMI  training  was  complete,  they  were  administered  a  posttest  of  their  knowledge. 
Finally,  the  Soldiers  completed  a  questionnaire  to  evaluate  their  learning  experience  with  the 
IMI,  and  a  small  number  were  briefly  interviewed  after  turning  in  their  questionnaire. 

Findings: 

Although  we  believed  scores  would  increase  for  all  IMI  conditions,  we  hypothesized  that 
Soldiers  assigned  to  the  tailored  training  IMI  conditions  would  have  the  greatest  increase  in 
scores  from  pretest  to  posttest  compared  to  familiarization  and  core/refresher.  Additionally,  we 
hypothesized  that  tailored  training  would  be  more  effective  for  Soldiers  receiving  training  on  a 
less  familiar  topic  domain  (i.e.,  Adjust  Indirect  Fire)  compared  to  a  more  familiar  topic  (i.e., 
Conduct  a  Defense  by  a  Squad)  because  the  tailored  training  IMI  provided  Soldiers  with  a 
diagnostic  assessment  and  individualized  feedback  to  assist  them  in  more  effectively  selecting 
content  on  which  to  focus  their  effort. 

As  anticipated,  there  was  a  statistically  significant  increase  in  scores  (i.e.,  8  to  12 
percentage  points)  across  all  IMI  variations.  However,  in  the  more  familiar  topic  domain,  the 
tailored  training  IMI  did  not  produce  a  significantly  greater  increase  in  scores  compared  to 
familiarization  and  core/refresher.  In  contrast,  for  the  less  familiar  topic  domain,  there  was  a 
statistically  significant  18  percentage  point  gain  between  pretest  and  posttest  scores  for  the 
tailored  training  variation.  This  finding  indicates  that  there  may  be  some  advantage  in  providing 
a  diagnostic  assessment  and  individualized  feedback  to  Soldiers  before  training  when  they  are 
learning  about  a  topic  with  which  they  are  less  familiar.  That  said,  in  the  more  familiar  content 
domain,  design  features  shared  across  all  IMI  variations — such  as  chunked  content — may  have 
facilitated  already  knowledgeable  Soldiers  in  tailoring  their  own  learning  experience  regardless 
of  IMI  condition. 

Utilization  and  Dissemination  of  Findings: 

This  report  presents  our  findings  from  the  final  phase  of  the  Tailoring  Multimedia 
Instruction  to  Soldier  Needs  research  effort.  The  results  of  the  experiment  were  briefed  to  MCoE 
DOTD,  4  November  2013,  were  presented  as  an  information  paper  to  the  Henry  Caro 
Noncommissioned  Officer  Academy  at  Fort  Benning,  GA  in  December  2013  and  to  the  Institute 
for  Noncommissioned  Officer  Professional  Development,  March  2014.  The  results  were  also 
presented  at  the  Interservice/Industry  Training,  Simulation  and  Education  Conference  in 
Orlando,  FL,  December  2014. 
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A  Comparison  of  Interactive  Multimedia  Instruction 
Designs  Addressing  Soldiers’  Learning  Needs 

INTRODUCTION 

The  Army  Learning  Model  (ALM)  has  emphasized  the  need  for  instruction  that  is  both 
tailored  to  career- specific  and  individual  learning  needs,  and  accessible  to  learners  where  and 
when  they  need  it  in  a  career-long  continuum  (TRADOC,  201 1).  In  pursuing  these  goals,  the 
Army  has  found  computer-based  interactive  multimedia  instruction  (IMI)  to  be  a  cost-effective 
way  to  provide  instruction  to  large  audiences  of  learners.  Moreover,  IMI  can  also  be  designed  to 
deliver  instruction  that  targets  the  learning  needs  of  particular  audiences  and  individuals.  Our 
research  focused  on  the  question  of  how  to  design  IMI  to  best  meet  Soldiers’  career- specific  and 
individual  learning  needs. 

This  research  report,  which  is  the  third  in  a  series  of  three,  documents  an  experiment 
conducted  to  evaluate  three  needs-focused  IMI  design  variations  for  two  content  domains. 

Earlier  phases  of  the  research  were  presented  in  two  previous  reports  (see  Blankenbeckler, 
Graves,  &  Wampler,  2013;  2014).  The  2013  report  presented  a  survey  of  existing  Army  Combat 
Arms  IMI  (e.g.,  Infantry,  Air  Defense,  Armor,  and  Field  Artillery).  This  survey  was  conducted 
to  identify  whether  and  how  existing  IMI  could  be  modified  to  various  point  of  need  focused 
instructional  designs,  targeting  a  particular  audience  with  job- specific  and  individual  learning 
requirements  (i.e.,  Blankenbeckler  et  al.,  2013).  The  2014  report  then  described  the  process  and 
rationale  for  developing  six  IMI  exemplars,  applying  various  principles  of  instructional  design  to 
address  the  learning  needs  of  new  squad  and  team  leaders  (i.e.,  Blankenbeckler  et  al.,  2014). 

The  experiment  documented  in  this  report  compared  the  effectiveness  of  three  types  of 
needs-focused  IMI  designs  (i.e.,  familiarization,  core/refresher,  and  tailored  training)  for  training 
a  particular  population  of  Soldiers  (E-4  Corporal/Specialist  and  E-5  Sergeants)  on  two  topics, 
one  more  familiar  to  them  (Conduct  a  Defense  by  a  Squad)  and  one  less  familiar  (Adjust  Indirect 
Fire).  The  results  of  this  experiment  may  be  used  to  inform  the  planning  and  development  of 
Army  IMI  focused  on  addressing  the  learning  needs  of  specific  audiences  and  individuals. 
Together,  the  three  reports  and  the  IMI  exemplars  address  Army  warfighting  challenges  to 
enhance  training  in  areas  of  tailored  distributed  training  and  education,  preparing  and  supporting 
Soldiers’  and  Leaders’  learning  at  their  point  of  need  (see  TRADOC,  2014). 

Background  on  the  Research  Problem 

Various  characteristics  of  IMI  work  together  to  shape  the  learning  experience  both  in 
terms  of  how  the  learning  content  is  presented — audio,  visual,  textual — and  how  learners  are 
able  to  interact  with  aspects  of  the  instructional  design.  Interactivity  can  be  as  simple  as  a 
learner  controlling  how  quickly  they  move  through  an  IMI  module  to  very  complex 
implementations  of  adaptive  or  tailored  training,  intended  to  create  individualized  learning 
experiences.  Although  interactivity  can  be  beneficial  to  learning,  too  much  interactivity  could 
compromise  learning  outcomes. 
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While  there  are  many  possible  ways  to  design  IMI,  it  has  to  be  relatively  self-contained 
and  self-correcting.  To  contrast  IMI  with  classroom  instruction,  an  instructor  in  a  classroom  can 
help  learners  to  recognize  problems  they  are  having  and  guide  them  in  seeking  effective 
solutions.  This  is  more  difficult  in  a  distance  learning  situation  which  means  IMI  should  be 
designed  to  assist  learners  in  making  reasonable  decisions  about  how  to  pursue  their  learning 
objectives,  even  if  they  start  out  with  little  awareness  of  what  they  need  to  learn. 

With  the  right  information  technologies  in  place,  IMI  can  reach  a  diverse  audience  of 
learners.  However,  the  IMI  made  available  to  this  audience  may  not  always  address  the  varying 
knowledge,  skills,  and  experiences  represented  among  its  members.  IMI  should  be  designed  to 
address  individual  differences  among  learners,  and  to  ensure  that  content  aligns  with  career  and 
professional  development  goals.  The  ALM  recognized  that  Soldiers  arrive  for  training  with  a 
variety  of  educational  backgrounds  and  professional  experiences,  and  these  differences  are 
reflected  in  different  learning  needs  (TRADOC,  2011).  Soldiers,  as  a  consequence,  require  IMI 
that  can  address  their  individual  learning  needs  as  well  as  shared  needs  specific  to  their  career 
path  and  career  point. 

Needs-Focused  IMI  Designs  for  Targeted  Audiences 

The  Army  has  developed  IMI  that  applies  adaptive  and  tailored  training  techniques  to 
address  Soldiers’  individualized  learning  needs  (see  review  in  Blankenbeckler  et  al.,  2013). 

Some  existing  Army  IMI  has  used  high-bandwidth  streaming  video  with  a  branching  story 
structure,  or  complex  algorithms  to  track  and  customize  Soldiers’  progress  through  the 
instructional  content.  Although  a  number  of  these  IMI  are  of  very  high  quality,  they  were  likely 
expensive  to  develop.  The  expense  involved  in  very  complex  design  features  may  deter  their  use 
in  IMI  designed  for  smaller,  more  specialized  audiences.  For  instance,  implementing  a  technique 
like  computer  adaptive  testing  (CAT)  requires  data  collected  from  a  large  number  of  participants. 
Without  that  sufficient  pool  of  learners  from  which  to  gather  data,  the  CAT  technique  would  be 
inappropriate.  Given  the  benefits  of  providing  training  tailored  to  Soldiers’  unique  learning 
needs,  it  is  necessary  to  determine  the  validity  of  simplified  tailoring  techniques  that  can  be 
applied  in  IMI  intended  for  smaller  audiences. 

Our  previous  research  explored  IMI  designs  to  address  the  learning  needs  of  smaller, 
more  specialized  audiences  (i.e.,  Blankenbeckler  et  al.,  2014).  In  this  research  effort,  we  sought 
to  empirically  validate  simplified  tailored  training  techniques  intended  to  address  Soldiers’ 
individual  and  career-specific  learning  needs.  Three  types  of  needs-focused  IMI  designs  were 
used  to  design  IMI  for  two  topics.  Both  topics  and  content  were  selected  to  target  the  learning 
needs  of  new  squad  and  team  leaders  (i.e.,  Specialist/Corporal,  E-4  and  Sergeant,  E-5  levels). 

The  training  topics  were  Adjust  Indirect  Fire  (Task  #061-283-6003)  and  Conduct  a  Defense  by  a 
Squad  (Task  #071-430-0002)  (U.S.  Army,  2008).  The  three  IMI  designs  included  one  tailored 
training  variation  and  two  non-tailored  training  variations,  labeled  familiarization  and 
core/refresher. 

Familiarization  was  a  non-tailored  IMI  design,  intended  to  provide  a  broad  overview  of 
content.  Through  familiarization  IMI,  the  learner  would  acquire  a  coherent  general 
understanding  of  the  training  topic  without  going  into  great  depth.  Much  of  the  existing  Army 
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IMI  follows  this  design  strategy  (Blankenbeckler  et  al.,  2013).  The  second  non-tailored  IMI 
design  was  core/refresher  training,  which  focused  on  providing  in-depth  information  on  how  to 
execute  a  specific  task.  As  a  result,  core/refresher  has  less  coverage  of  breadth  information  (i.e., 
contextual  content)  than  familiarization.  This  design  strategy  was  selected  on  the  basis  of 
discussion  with  MCoE  DOTD,  who  expressed  a  need  to  utilize  multimedia  training  designs  that 
provide  for  rapid,  in-depth  refresher  training.  Finally,  the  tailored  training  design  combined  both 
the  breadth  of  familiarization  and  the  depth  of  core/refresher  training  into  multiple,  learner- 
selected  training  paths,  each  providing  different  content.  In  the  tailored  condition,  a  diagnostic 
assessment  with  individualized  feedback  was  provided  to  learners  before  and  after  training  to 
help  inform  their  decisions  to  pursue  a  particular  path  through  the  instructional  content.  Rather 
than  automated  selection,  learners  selected  their  own  instructional  path  from  among  those 
available.  Of  the  three  designs  tested,  two  were  non-tailored,  and  one  was  tailored;  the  two  non- 
tailored  designs  differed  in  terms  of  the  breadth  versus  depth  of  content  covered  whereas  the 
tailored  training  design  incorporated  both  depth  and  breadth  of  content. 

Why  Three  IMI  Variations:  The  Learner  Autonomy  Concept 

A  central  concept  in  the  ALM  is  that  of  learner-centered  training.  In  learner-centered 
training,  the  focus  is  both  on  providing  training  that  addresses  individual  learning  needs  as  well 
as  increasing  the  responsibility  of  learners  for  their  own  success.  In  the  learning  research 
literature,  the  concept  of  learner  autonomy  is  associated  with  ongoing  debate,  but  a  widely  cited 
definition  refers  to  the  “capacity  to  take  control  over  one’s  own  learning,”  with  the  goals  of 
instructional  institutions  to  improve  learners’  ability  to  learn  on  their  own  and  to  facilitate  access 
to  required  learning  materials  (Benson,  2011,  pg.  2).  However,  simply  providing  learners  with 
more  materials  they  can  use  when  learning  on  their  own  (i.e.,  using  more  distance  learning 
technologies)  does  not  necessarily  lead  them  to  become  more  autonomous  learners.  Therefore, 
the  learning  materials  and  selected  content  needs  to  be  designed  to  facilitate  the  learners’  role  in 
the  learning  process. 

There  are  gradations  of  learner  autonomy  (see  Benson,  201 1).  One  can  consider  fully 
autonomous  learners  to  be  those  who  are  selecting  their  own  learning  materials,  pacing  their 
learning  process,  self-assessing,  and  determining  how  to  proceed  throughout  the  learning 
process.  While  self-directed  learning  of  this  type  is  a  very  complex  process,  it  may  also  be  a 
widespread  practice  among  adult  learners  in  the  Army  (see  Graves,  Rauchfuss,  &  Wisecarver, 
2012).  Fully  autonomous  learning  requires  a  learner  who  is  already  somewhat  knowledgeable 
about  a  domain  and  has  the  requisite  skills  to  plan  and  execute  a  self-guided  learning  process 
(Dickenson,  1987). 

There  are  also  situations  in  which  it  is  appropriate  for  learners  to  be  semi-autonomous. 
Instruction  can  be  designed  to  structure  the  learning  experience,  but  the  learner  can  also  be  an 
active  participant  by  selecting  among  available  pathways  through  the  learning  content.  Feaming 
materials  can  be  designed  to  guide  learners  during  the  learning  process,  even  if  learners  are  not 
yet  knowledgeable  about  the  domain  or  are  not  yet  skilled  at  self-guided  learning.  The  IMI  we 
designed  in  this  effort  sought  to  address  different  levels  of  learner  autonomy,  with  familiarization 
and  core/refresher  intended  to  support  lower  levels  of  learner  autonomy  and  tailored  training  to 
support  a  higher  level  of  autonomy. 
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Multimedia  Design  Decisions 


Instructional  designers  create  learning  environments  to  promote  learning,  adjusting 
design  strategies  to  varying  levels  of  learner  control  and  autonomy  (Mayer,  2008).  In  doing  so, 
they  must  determine  how  best  to  convey  ideas,  selecting  among  possibilities  such  as  speech, 
written  words,  illustrations,  charts,  or  videos  (Mayer  &  Moreno,  2003).  With  each  design 
choice,  the  designer  determines  what  particular  types  of  materials  will  be  used  and  what  is  the 
best  way  to  organize  content  (Mayer,  2008).  For  multimedia  instructional  design,  it  may  be 
useful  to  think  in  terms  of  how  language  (written  and  spoken)  and  pictures  may  be  most 
effectively  used  together  for  instructional  purposes  within  a  particular  medium  (see  Bartholome 
&  Bromme,  2009;  Mayer  &  Moreno,  2003).  Moreover,  it  is  useful  to  consider  the  ways  in  which 
learners  make  sense  of  multiple  types  of  information  being  presented  to  them  through  multiple 
sensory  modalities. 

The  purpose  of  IMI  is  to  encourage  learners  to  actively  integrate  information  (Gyselinck, 
Jamet,  &  Dubois,  2008).  For  learners  to  actively  integrate  information,  it  is  up  to  the 
instructional  designer  to  make  various  segments  of  instruction  work  together,  both  in  terms  of 
spatial  layout  of  text  and  images,  as  well  as  the  sequence  and  timing  of  how  information  is 
presented.  For  instance,  a  designer  could  provide  learners  with  information  that  is  sequenced 
from  familiar  to  unfamiliar.  This  approach  may  help  learners  build  a  more  complex 
understanding  by  activating  prior  knowledge  and  providing  a  context  for  learning  (Mayer,  2001; 
Mayer  &  Moreno,  2003).  When  instruction  is  designed  in  this  way,  learners  may  be  better 
prepared  to  make  meaningful  connections  between  the  concepts  they  are  learning  and  what  they 
already  know.  Moreover,  by  making  prior  knowledge  salient  before  introducing  new  concepts, 
learners  may  also  be  better  positioned  to  recognize  general  principles  that  could  support  transfer 
of  what  they  learned  to  other  contexts  (Mayer  &  Moreno,  2003). 

Research  has  suggested  that  IMI  is  effective  due  to  particular  design  choices  and  the 
degree  to  which  learners  are  encouraged  to  actively  shape  their  learning  experiences  (Lusk  et  al., 
2009).  Planning  for  interactivity  is  important,  and  can  be  as  simple  as  allowing  learners  to 
control  how  quickly  they  progress  through  the  materials  (Mayer  &  Chandler,  2001).  Designing 
IMI  to  support  varying  degrees  of  learner  autonomy  may  have  benefits  for  learning  outcomes 
(Mayer,  2008). 

Instructional  Design  Principles  and  IMI 

Organizations  recognize  benefits  from  IMI,  such  as  being  able  to  reach  a  large  audience 
with  compelling  learning  materials,  centralizing  control  of  learning  materials,  and  cost  savings 
from  bringing  the  learning  environment  to  the  student  rather  than  the  student  to  the  environment 
(Fletcher,  2009).  Like  any  form  of  instruction,  however,  quality  is  ultimately  determined  by 
content  selection  and  careful  design.  Instructional  design  needs  to  support  learners  in  making 
sense  of  what  they  are  learning.  This  recognition  has  motivated  research  on  the  ways  in  which 
instructional  design  intersects  with  human  cognitive  processes,  focusing  on  topics  such  as  how 
learners  perceive  and  make  sense  of  relevant  information,  remember  critical  information,  or 
apply  what  they  learned  to  solve  problems  and  make  decisions  (Schiller,  Scheiter,  &  van 
Genuchten,  2011). 
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Researchers  have  experimented  with  various  instructional  design  features  (e.g., 
highlighting  text,  structure  of  content,  or  varying  communication  style)  and  compared  the 
learning  outcomes.  They  have  prescribed  guidelines  to  use  when  designing  IMI  (see  Clark  & 
Mayer,  2008).  Mayer  (2005)  has  argued  that  the  most  effective  instructional  designs  reduce 
irrelevant  information  and  help  learners  to  think  more  deeply  about  the  training  content.  This  is 
done  by  (a)  reducing  extraneous  cognitive  processing,  (b)  managing  essential  cognitive 
processing,  and  (c)  facilitating  generative  processing. 

To  reduce  extraneous  cognitive  processing,  instructional  designs  should  remove 
distracting  information  and  emphasize  critical  information.  This  can  be  done  through  careful 
selection  of  words  and  images,  offsetting  or  highlighting  critical  ideas,  avoiding  putting  too 
much  information  in  one  place,  using  visual  space  logically,  and  pacing  instruction  carefully. 
Applying  these  techniques  may  help  learners  to  identify  relevant  content  on  which  to  focus  their 
time  and  effort. 

To  manage  essential  cognitive  processing,  the  instruction  should  be  designed  to  help 
learners  make  sense  of  what  they  are  learning.  This  design  goal  focuses  on  interrelated 
principles  that  allow  learners  to  self-pace  their  learning,  make  use  of  preexisting  knowledge,  and 
reduce  conflicts  between  sensory  modalities.  For  example,  a  designer  should  avoid  presenting 
animations  along  with  static  text  because  both  require  learners’  visual  attention. 

Finally,  to  facilitate  generative  processing  design  features  should  be  incorporated  to 
support  learners  in  achieving  a  deeper  understanding  of  the  material  they  are  learning.  The  goal 
is  to  reduce  as  much  as  possible  any  barriers  to  understanding  by  establishing  an  appropriate 
communication  style  with  a  learner  (see  Clark  &  Mayer,  2008;  Mayer,  2009;  Moreno  &  Mayer, 
2007).  Table  1  presents  various  design  features  and  how  they  have  been  applied  in  IMI. 

Contributions  of  Cognitive  Psychology  to  Instructional  Design 

A  scientific  understanding  of  human  learning  capabilities  and  needs  should  set 
boundaries  for  instructional  design  (Tardieu  &  Gyselinck,  2003).  All  learners  have  capacities 
that  limit  how  much  and  how  quickly  they  can  perceive,  attend  to,  understand,  recall,  and  use 
newly  acquired  information.  Learners  also  differ  in  terms  of  their  mental  abilities,  background 
experiences,  and  prior  knowledge.  Bearing  this  in  mind,  IMI  must  be  designed  to  support 
learning  within  these  normal  human  limitations.  Research  in  cognitive  psychology — which 
looks  at  how  human  beings  think,  remember,  plan,  decide,  problem-solve,  and  learn — provides  a 
useful  account  of  the  ways  in  which  learners  can  differ.  By  focusing  on  human  cognition,  one 
may  find  a  way  toward  more  effective  instructional  designs. 
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Table  1 


IMI  Design  Features  and  their  Application 


Goal 

Features 

Application 

To  Reduce 
Extraneous 

Coherence 

Eliminating  extraneous  words,  pictures,  images 

Cognitive 

Processing 

Signaling 

Highlighting  important  words  (e.g.,  section  headings, 
highlighting,  boldface  font) 

Redundancy 

Combining  animations  with  narrations  rather  than 
animation,  narration  and  text 

Spatial  Contiguity 

Placing  corresponding  portions  of  pictures  and  words  near 
each  other 

Temporal  Contiguity 

Presenting  corresponding  animation  and  narration 
simultaneously  rather  than  successively 

To  Manage 
Essential 

Segmenting 

Presenting  narrated  animation  in  learner-paced  segments 

Cognitive 

Processing 

Pre-training 

Providing  pre-training  in  vocabulary  and  key  concepts 
(e.g.,  outlines,  key  learning  objectives,  bottom  line  up 
front) 

Modality 

Combining  animation  (visual)  with  narration  (auditory), 
not  animation  (visual)  with  text  (visual) 

Guided  Activity 

Prompting  learners  to  select,  organize,  and  integrate  new 
information 

Reflection 

Encouraging  self-reflection  to  activate  organization  and 
integration  of  new  information 

Feedback 

Providing  learners  with  proper  schemas  to  repair 
misconceptions 

Worked  Examples 

Leveraging  worked  examples  to  show  how  to  work  though 
tasks/problems  step-by-step 

To  Encourage 
Generative 

Personalization 

Communicating  in  an  informal/conversational  style 

Processing 

Voice 

Narrating  in  a  non-accented  voice  rather  than  a  machine- 
simulated  voice 

Pacing 

Allowing  learners  to  control  their  pace,  and  process 
smaller  chunks  of  information  in  working  memory 

Sequencing 

Ordering  information  to  move  from  old  (familiar) 
information  to  new  (unfamiliar)  information 

Clear  Structure 

Using  a  familiar  structure/pattern  for  presenting 
information  (e.g.,  compare-contrast,  classification, 
enumeration,  cause-effect) 

Note:  Table  adapted  from  Blankenbeckler  et  al.  (2014). 


Working  Memory 

Memory  is  an  essential  cognitive  process  for  learning.  Being  able  to  remember  is  one 
way  we  intuitively  determine  whether  we  have  learned.  Cognitive  psychologists  have  shown, 
however,  that  remembering  is  a  complex  process  with  many  limitations.  Evidence  suggests  that 
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there  may  be  several  memory  systems  involved  in  how  we  learn.  For  instance,  one  well-known 
model  identifies  three  interacting  memory  systems:  auditory,  visual,  and  executive  (see  Baddeley 
&  Hitch,  1974).  In  emphasizing  the  role  of  the  executive  system,  this  model  of  memory  suggests 
that  getting  information  into  memory  is  more  than  a  matter  of  just  repeatedly  encountering  it. 
Information  needs  to  be  encoded  in  such  a  way  that  it  is  meaningful  to  learners.  This  encoding 
process  is  particularly  important  for  adult  learners,  who  bring  far  more  contextual  knowledge  and 
experience  to  the  learning  situation  compared  to  children.  By  understanding  what  the  audience 
likely  already  knows,  designers  can  plan  how  they  will  introduce  new  knowledge  by  making 
effective  use  of  these  different  memory  systems. 

Attention 

Attention  is  another  cognitive  process  important  for  memory.  How  we  attend  to  what  we 
see  and  hear  can  affect  how  we  later  remember  it  (Baddeley,  2000).  Attention  can  be 
consciously  controlled.  Even  though  our  immediate  auditory  and  visual  memories  have  a  very 
brief  span,  they  do  retain  information  long  enough  that  we  can  begin  to  make  sense  of  what  we 
have  perceived  (Sperling,  1960;  Neisser,  1967).  For  instance,  consider  the  experience  of 
someone  unexpectedly  talking  to  you  while  you  are  concentrating  on  another  task.  You  may 
hear  sounds,  but  it  is  only  after  you  shift  your  focus  to  those  sounds  that  it  may  take  an  additional 
second  or  so  before  you  understand  what  was  said.  That  shift  in  attention  is  possible  because 
auditory  memory  can  retain  the  sounds  of  the  words  that  were  spoken  in  a  raw  state,  even  though 
you  were  not  paying  attention  to  them.  Auditory  memory  tends  to  be  longer  (2  to  4  seconds) 
than  visual  memory  (less  than  1  second),  however,  both  systems  function  to  hold  raw 
information  long  enough  that  we  can  make  sense  of  it.  In  experimental  psychology,  a  common 
laboratory  task  is  to  have  research  participants  learn  information  with  other  information  present 
to  distract  them  and  interfere  with  the  learning  process.  While  interference  is  excellent  for 
isolating  mental  processes  in  a  laboratory,  it  detracts  from  instructional  design.  One  should 
consider  the  types  of  actions  the  learner  is  being  asked  to  take  when  interacting  with  the  media, 
such  as  searching  for  information  or  making  mental  calculations,  and  avoid  conflicts  in  what 
learners  are  being  asked  to  pay  attention  to  (Gyselinck,  Ehrlich,  Comoldi,  de  Beni,  &  Dubois, 
2000). 


Cognitive  psychology  can  inform  the  work  of  instructional  designers.  It  provides  a  way 
to  understand  why  certain  types  of  multimedia  instructional  designs  may  be  more  effective  or 
less  effective  for  learning.  When  engaged  in  multimedia  learning,  learners  have  to  integrate  both 
visual  and  auditory  information  into  a  coherent  message  (Reed,  2006).  If  the  information  is 
being  presented  too  quickly,  or  through  too  many  modalities,  the  executive  function  of  memory 
can  become  overwhelmed.  When  this  occurs  during  learning,  learners  will  either  slow  down  or 
begin  to  miss  critical  information,  limiting  their  ability  to  make  sense  of  the  content  and  connect 
it  with  information  they  already  know. 

Moreover,  how  information  is  processed  through  different  sensory  modalities  should  be 
considered.  Some  research  suggests  that  there  is  benefit  to  receiving  information  through 
different  sensory  modalities,  as  long  as  it  is  not  being  presented  simultaneously  to  the  same  or 
similar  modalities  (Clark  &  Paivio,  1991).  For  instance,  when  reading  text,  some  people  will 
hear  the  words  they  are  reading;  others  may  make  sense  of  the  words  in  terms  of  what  they  are 
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seeing  (see  Schuler  et  al.,  201 1).  Whether  we  hear  or  see  words  while  reading  implies  that 
different  memory  systems  may  have  been  called  on  to  process  that  information.  For  learners 
who  hear  as  they  read,  the  narration  within  IMI — if  it  is  presented  with  visual  text — can  hinder 
their  ability  to  understand.  The  same  memory  system  is  receiving  two  potentially  conflicting 
streams  of  information  thereby  forcing  the  executive  system  to  focus  on  one  stream  of 
information,  or  else  both  may  become  jumbled  together  (Baddeley,  1986;  2000).  This  suggests 
that  careful  sequencing  of  new  information  relying  on  different  modalities  may  be  useful  in 
reducing  cognitive  processing  conflicts  and  support  learners  in  forming  a  coherent  understanding 
of  what  they  are  learning  (Bruner,  1990). 

Extrinsic,  Intrinsic,  and  Germane  Cognitive  Load 

Cognitive  load  describes  the  mental  effort  expended  by  working  memory,  and  may 
provide  a  framework  for  understanding  how  cognitive  processes  and  instructional  design  are 
related  (Paas,  Renkl,  &  Sweller,  2003).  Cognitive  load  theory  enables  researchers  to  identify 
sources  of  load  imposed  during  learning,  and  determine  how  these  sources  of  load  may  interact 
to  influence  learning  outcomes  (Schuler  et  al.,  2011). 

There  are  three  types  of  sources  for  cognitive  load:  extrinsic,  intrinsic,  and  germane  load. 
Extrinsic  load  arises  from  the  structure  and  complexity  of  the  learning  materials  (Briinken,  Plass, 
&  Leutner,  2003).  For  instance,  poorly  designed  IMI  may  impose  unnecessary  extrinsic  load 
because  of  its  inherent  structure  and  not  due  to  the  difficulty  of  content  (Paas  et  al.,  2003). 
However,  the  imposition  of  extrinsic  load  can  be  well-intentioned.  Presenting  written  text  that  is 
verbatim  to  narrated  content — a  misguided,  yet  common  practice — can  at  times  impose 
unnecessary  load  by  splitting  learners’  attention  (Mayer,  2001).  Repeating  information  in 
multiple  presentation  modalities  is  often  unnecessary  if  learners  are  afforded  the  option  of 
replaying  sections  of  narration.  Therefore,  IMI  instructional  designs  should  focus  on  reducing 
extrinsic  load  as  much  as  possible  to  support  efficient  learning. 

Intrinsic  load  refers  to  the  cognitive  effort  required  to  integrate  new  information  with 
prior  knowledge  (Moreno  &  Park,  2010).  It  is  far  more  difficult  to  reduce  intrinsic  load  as 
effectively  as  extrinsic  load  because  some  content  domains  are  more  difficult  to  learn.  The 
difficulty  of  content  can  affect  instructional  designers’  ability  to  reduce  intrinsic  load,  especially 
for  learners  who  are  new  to  a  content  domain.  However,  one  of  the  ways  to  reduce  this  load  is 
through  application  of  tailored  training  principles  that  can  help  to  shape  instruction  to  individual 
learners’  prior  knowledge  and  experience  (Pass  et  al.,  2003;  Spanjers,  Wouters,  van  Gog,  &  van 
Merrienboer,  2011). 

That  said,  for  learners  who  are  more  familiar  with  a  content  domain,  their  preexisting 
understanding  of  that  domain  can  reduce  their  intrinsic  load  while  learning.  Having  prior 
knowledge  and  experience  may  help  to  improve  integration  of  new  information  for  learners  who 
are  familiar  with  a  domain.  Moreover,  by  having  a  reduced  intrinsic  load,  learners  who  are 
familiar  with  a  domain  may  be  better  positioned  to  address  extraneous  sources  of  load 
(Khacharem  et  al.,  2013;  Spanjers  et  al.,  201 1).  In  other  words,  learners  familiar  with  a  domain 
may  more  effectively  use  learning  materials  of  varying  complexity  and  quality  than  learners  who 
are  less  familiar  with  a  domain. 
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Finally,  germane  load  refers  to  cognitive  resources  that  the  learner  focuses  on  to  construct 
a  mental  model  and  understand  the  content  domain  (Sweller,  2010).  For  instance,  by  providing 
well-structured  and  well-organized  training,  with  clearly  labeled  sections  and  subsections,  the 
learner  may  be  exposed  to  a  coherent  structure  of  the  domain.  This  ordered  structure  could  help 
them  organize  information  in  a  way  that  will  be  effective  for  them  in  developing  a  mental  model. 
In  addition,  tools  to  assist  learners  in  planning  in  advance  how  they  will  work  through  the 
content  can  mitigate  difficulties  they  might  otherwise  encounter  while  learning  from  complex 
materials. 

Prior  Knowledge  and  Experience 

Prior  knowledge  and  experience  may  strongly  affect  how  well  a  learner  can  learn  from  a 
particular  IMI,  even  when  design  features  have  been  incorporated  to  mitigate  sources  of 
cognitive  load  (Briinken  et  al.,  2003;  Kalyuga,  Ayres,  Chandler,  &  Sweller,  2003).  One  example 
of  this  is  the  expertise  reversal  effect.  The  expertise  reversal  effect  occurs  when  certain  training 
designs  and  content  selections  may  be  associated  with  decreased  performance  for  knowledgeable 
learners.  From  a  design  standpoint,  the  effect  is  typically  associated  with  too  much  basic 
information  being  provided  to  learners  who  already  have  a  strong  understanding  of  the  domain 
(Kalyuga,  2009).  Some  instructional  designs  appropriate  for  learners  with  less  prior  knowledge 
become  inappropriate  as  learners  become  more  knowledgeable  in  a  domain,  especially  when 
designs  lockstep  advanced  learners  through  basic  content  (Kalyuga  &  Renkl,  2010).  Therefore, 
when  possible,  it  is  advantageous  to  evaluate  the  prior  knowledge  of  target  audiences,  using 
techniques  like  front-end  or  needs  analysis,  before  designing  IMI  in  order  to  a)  promote  learning 
and  b)  avoid  wasting  time  and  resources  pursuing  a  potentially  ineffective  design  strategy  (see 
Yardakul,  Uslu,  Cakar,  &  Yildiz,  2014). 

Applications  of  Effective  Design  Principles 

A  number  of  IMI  design  concepts  arise  from  a  scientific  understanding  of  working 
memory,  attention,  cognitive  load,  and  prior  knowledge  and  experience.  The  following  design 
principles  can  be  incorporated  into  IMI  to  increase  learner  engagement  and  to  facilitate  learner 
autonomy,  and  include  the  multimedia  principle,  modality  principle,  redundancy  principle, 
segmentation  principle,  and  signaling/cueing. 

Multimedia  Principle 

Fundamental  to  IMI  is  the  idea  that  learning  is  more  effective  when  information  is 
presented  in  multiple  formats.  That  is,  learners  may  learn  better  from  combinations  of  words  and 
pictures  than  from  words  alone.  Research  has  indicated  that  IMI  that  encourages  learners  to 
connect  different  sources  of  information  tends  to  be  more  effective  (Mayer,  2002).  Most 
multimedia  learning  environments  combine  text  and  pictures  (Bartholome  &  Bromme,  2009). 
These  may  range  from  using  imagery  from  static  stills  (i.e.,  diagrams,  charts,  photos,  etc.)  to 
dynamic  animations  (Mayer,  2011).  Images  can  encourage  in-depth  processing  by  providing 
ready  visual  representations  of  spoken  or  written  words  (Larkin  &  Simon,  1987),  reducing 
cognitive  load  (van  Genuchen,  Scheiter,  &  Schuler,  2012)  and  supporting  learners  in  making 
inferences  about  what  they  are  learning  (Schmidt- Weigand  &  Scheiter,  2011).  Incorporating  text 
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and  pictures  can  encourage  better  integration  of  new  information  with  prior  knowledge  (Mayer, 
2002;  2008).  Finally,  pictures  and  diagrams  may  be  beneficial  for  learning  procedural  tasks  (van 
Genuchen  et  al.,  2012),  that  is,  tasks  which  have  goals  that  can  be  achieved  by  executing  a  series 
of  specific  actions. 

Modality  Principle 

Learners  process  visual  and  verbal  information  with  separate  cognitive  systems  (see 
Baddeley,  1992;  Paivio,  1986).  Presenting  too  much  information,  either  verbally  or  visually,  can 
overload  learners’  information  processing  capabilities  (Briinken,  Plass,  &  Leutner,  2003). 
Therefore,  when  incorporating  the  multimedia  principle  into  designing  IMI,  designers  may 
benefit  from  also  applying  the  modality  principle  to  reduce  cognitive  load.  Research  has 
indicated  that  learners  are  more  successful  when  learning  from  animations  paired  with  narrations 
over  presentations  comprised  of  static  text  with  on-screen  images  (Moreno  &  Mayer,  1999). 
Presenting  static  text  with  on-screen  images  may  split  attention  between  relevant  sources  of 
information  presented  visually  (Schiller,  Scheiter,  Rummer,  &  Gerjets,  2012;  Mousavi,  Low,  & 
Sweller,  1995).  Therefore,  narrations  are  beneficial  because  they  enable  the  learner  to  attend  to 
the  visual  information  while  also  processing  audio  content,  thereby  reducing  conflict  between 
these  different  information  processing  systems  (Gyselinck  et  al.,  2008;  Kalyuga,  Chandler,  & 
Sweller,  2000).  Narration  with  images  enables  learners  to  process  complex  information 
simultaneously;  whereas,  text  with  images  requires  sequential  processing  (Rummer,  Schweppe, 
Fiirstenberg,  Seufert,  &  Briinken,  2010). 

Redundancy  Principle 

While  incorporating  redundant  information  into  lessons  may  facilitate  learning  because  it 
provides  multiple  opportunities  to  learn  content  (Mayer,  2001),  it  can  also  impair  learning  by 
violating  the  modality  principle  (Mayer,  Heiser,  &  Lonn,  2001).  In  one  study,  students  who 
viewed  narrated  animations  with  concurrent  visual  text  performed  significantly  worse  than 
students  who  viewed  only  narrated  animations  (Mayer  et  al.,  2001).  If  it  is  necessary  to  include 
redundant  information  within  IMI,  adding  a  few  redundant  words  from  a  narration  and 
presenting  them  visually  can  improve  learning  as  long  as  the  redundant  information  is  short, 
highlights  essential  concepts,  and  is  presented  in  close  proximity  to  the  portion  of  the 
graphic/animation  being  described  (Mayer  &  Johnson,  2008).  Novices  may  need  more  guidance 
when  learning  and  can  benefit  from  redundant  on-screen  texts,  whereas  experts  may  find  it 
distracting,  especially  when  they  cannot  navigate  around  the  redundancies  (see  Kalyuga  et  al., 
2003). 

Segmentation  Principle 

Long  sections  of  training  can  impose  a  higher  cognitive  load  (Mayer  &  Moreno,  2003). 

To  reduce  this  load,  segmentation  has  been  proposed  as  a  useful  instructional  principle  that  can 
be  employed  to  divide  longer,  continuous  instruction  into  shorter  segments  (Cheon,  Crooks,  & 
Chung,  2014;  Mayer  &  Moreno,  2003).  By  presenting  less  content  per  segment,  learners  may  be 
better  able  to  process  materials  without  experiencing  too  much  cognitive  load.  Although  learners 
can  benefit  from  system-controlled  segmented  presentations  (Spanjers  et  al.,  2011),  learners 
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benefit  more  when  they  control  the  pace  of  animations  (i.e.,  segmentation  principle;  Mayer  & 
Chandler,  2001).  Designing  IMI  to  incorporate  segmentation  requires  instructors  to  divide 
multimedia  tutorials  into  segments  (Doolittle,  2010).  At  the  end  of  each  segment,  the  animation 
stops.  At  that  point,  the  learner  then  decides  when  to  advance  to  the  next  segment  by  clicking  a 
“continue”  button.  By  allowing  learners  to  set  their  own  pace  of  learning,  segmentation  helps 
create  a  more  interactive  learning  environment  and  may  promote  a  better  transfer  of  learning 
(Mayer,  2002). 

Signaling/Cueing 

Learners  can  sometimes  have  difficulty  identifying  salient  information  (Boucheix,  Lowe, 
Putri,  &  Goff,  2013).  The  more  complex  an  IMI  presentation  becomes,  the  more  selective  the 
learners  tend  to  become  in  what  they  pay  attention  to  (Betrancourt,  2005).  Novice  learners  tend 
to  focus  on  perceptual  content  that  may  or  may  not  be  relevant  (Boucheix  et  al.,  2013).  Visual 
cueing  can  be  an  effective  way  to  get  learners  to  attend  to  critical  information  (Lin  &  Atkinson, 
201 1)  by  guiding  them  without  adding  to  their  cognitive  load  (Mayer  &  Moreno,  2003). 
Changing  colors  or  emphasizing  critical  text  has  been  shown  to  improve  immediate 
comprehension  (Boucheix  &  Guignard,  2005).  Likewise,  using  diagrams  with  color  and  arrows 
to  visualize  technical  concepts  also  produced  an  increase  in  learning  (Huk,  2006).  However, 
cueing  may  not  always  lead  to  better  learning  outcomes  (Kriz  &  Hegarty,  2007).  Cues  are  most 
effective  if  not  obscured  by  overly  complex  graphics  or  animations,  and  if  learners  need  such 
cues  to  support  their  learning  (de  Koning  et  al.,  2009).  If  cues  are  overused,  learners’  attention 
can  be  divided,  compromising  learning  outcomes  (Moreno,  2007). 

Summary  of  Design  Principles 

The  following  points  summarize  recommendations  from  the  literature  concerning  how  to 
optimize  IMI  design  to  address  learners’  needs: 

•  When  designing  and  developing  IMI  for  specific  audiences,  it  is  important  to  clearly 
define  up-front  what  learners  should  be  able  to  do  after  instruction.  Integral  to  this 
process  is  determining  the  baseline  levels  of  knowledge  and  experience  possessed  by  the 
target  audience. 

•  Connecting  content  with  the  knowledge  and  experience  of  a  particular  audience  enables 
the  designer  to  incorporate  elements  of  real-world  tasks  and  missions  into  the  IMI  content 
in  order  to  create  links  to  what  learners  will  be  expected  to  do  when  applying  what  they 
have  learned.  Providing  varied  examples  and  problems,  with  progressive  levels  of 
complexity  and  reduced  assistance  as  learners  become  more  familiar  with  the  content 
helps  to  maintain  their  independence,  motivation,  and  interest. 

•  Learners  may  benefit  from  being  provided  individualized  performance  feedback  and 
tailored  remediation. 

•  It  is  beneficial  to  provide  content  to  learners  in  smaller  logical  chunks.  Segmenting  the 
content  into  learner-paced  sections  helps  the  learner  to  manage  their  cognitive  load  by 
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giving  them  control  over  the  pacing  of  instruction  without  overloading  their  ability  to 
remember  what  they  are  learning. 

•  Providing  learners  with  pre-training,  covering  key  concepts  and  vocabulary,  can  be 
helpful  to  ensure  they  have  requisite  knowledge. 

•  It  is  easier  for  a  learner  to  make  sense  of  spoken  words  rather  than  text;  however,  the 
speech  style  of  the  narrator  should  be  intelligible  to  a  broad  audience,  and  use  a 
conversational  style.  It  is  better  to  use  a  human  narrator  rather  than  a  machine  voice. 

•  When  laying  out  the  content  on  the  screen,  it  is  important  to  highlight  only  the  most 
critical  information  to  ease  the  learning  experience.  Combining  on-screen  text  with 
narrated  animation  can  be  confusing  to  learners.  When  presenting  text  with  static 
images,  the  text  should  be  placed  close  to  the  image  it  describes. 

•  When  narration  is  used  to  describe  animation,  the  two  should  be  synchronized. 

•  Finally,  it  is  important  to  provide  individualized  feedback  to  learners,  such  as  providing  a 
diagnostic  assessment  to  support  remediation  prior  to  training  or  determination  of  an 
individualized  learning  path.  Assessments  can  also  be  provided  en-route,  following 
chunks  of  training  or  sections  that  cover  critical  skills.  An  end  of  training  assessment  can 
be  useful  to  determine  one’s  level  of  proficiency  or  to  identify  re-training  needs. 

Emphasizing  learner-centered  and  life-long  learning  initiatives,  the  ALM  views  IMI  as 
essential  to  delivering  instruction  where ,  when ,  and  how  it  is  needed  (TRADOC,  201 1).  For  IMI 
designers  to  ask  where  the  IMI  will  be  used  focuses  them  on  the  accessibility  afforded  by  various 
learning  technologies.  Asking  when  it  will  be  used  focuses  on  the  life-long  learning  needs  of  a 
target  audience.  Asking  how  it  will  be  used  focuses  on  the  needs  of  individual  learners.  The 
current  research  specifically  focused  on  the  when  and  how  questions — i.e.,  content  selection  and 
instructional  design.  Further,  the  impact  that  different  technologies  have  on  delivering  IMI  at  the 
point  of  need  is  recognized.  The  current  research  was  focused  on  exploring  how  instruction  can 
be  designed  to  meet  the  individual  learning  needs  of  new  Noncommissioned  Officers  (NCOs)  as 
they  are  preparing  to  become  squad  or  team  leaders.  This  current  research  compared  variations 
in  needs-focused  IMI  instructional  design  and  different  types  of  content  selections  to  target  the 
needs  of  a  specific  audience  of  learners. 


METHOD 

Three  needs-focused  IMI  designs  were  compared  for  two  topics  related  to  Combat  Arms. 
One  topic  was  more  familiar  to  learners  (i.e.,  Conduct  a  Defense  by  a  Squad),  and  one  topic  was 
less  familiar  (i.e.,  Adjust  Indirect  Fire).  The  selected  IMI  instructional  content  targeted  the 
learning  needs  of  new  squad/team  leaders  (E-4  Specialist/Corporal,  and  E-5  Sergeant  ranks). 

The  IMI  were  experimentally  tested  with  Soldiers  enrolled  in  the  Warrior  Leader  Course  at  Fort 
Benning,  GA. 
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Materials 


In  developing  the  six  IMI  exemplars  for  the  experiment,  the  researchers  made  use  of 
some  existing  Army  IMI  collected  during  the  first  phase  of  this  research  (see  Blankenbeckler  et 
al.,  2013  for  a  more  detailed  discussion),  but  for  the  most  part  developed  new  content.  Existing 
IMI  material  had  been  collected  from  the  Directorates  of  Training  and  Doctrine  at  the  Maneuver 
Center  of  Excellence  (Fort  Benning,  GA),  the  Fires  Center  of  Excellence  (Fort  Sill,  OK),  and  the 
Iowa  National  Guard  Distributed  Learning  Development  Center  (Camp  Dodge,  IA).  Also 
collected  were  examples  of  IMI  from  the  Army  Knowledge  Online  My  Training  Tab  (MT2)  site. 

Parts  of  the  existing  IMI  were  selected  to  be  reused  in  the  experimental  IMI  according  to 
the  following  criteria:  (a)  the  content  suitable  for  the  intended  audience,  i.e.,  new  squad  or  team 
leaders,  (b)  the  original  source  files  were  available  and  editable,  (c)  source  materials  were 
compatible  with  current  software  available  to  modify  files,  (d)  the  concepts,  uniforms, 
equipment,  and  materials  depicted  were  current,  and  (e)  the  instructional  design  characteristics  of 
the  existing  IMI  were  compatible  with  the  planned  modifications  (e.g.,  content  selection;  quality 
of  graphics,  animation,  text,  and  narration;  navigation/organization). 

Initially,  the  researchers  had  intended  to  produce  exemplars  of  needs-focused  IMI  based 
entirely  on  existing  content.  Ultimately,  only  30%  of  the  existing  materials  were  reusable  in  the 
way  intended  (see  Blankenbeckler  et  al.,  2013).  Most  of  the  existing  IMI  lacked  source  files  that 
could  be  modified  or  had  been  designed  in  superseded  versions  of  software.  In  addition,  much  of 
the  existing  IMI  was  designed  to  address  a  general  audience,  lacking  the  details  necessary  to 
focus  content  on  squad  or  team  leaders’  specific  learning  needs.  Our  experience  was  similar  to 
that  reported  by  other  Government  and  private  sector  organizations  (see  Shanley,  Lewis,  Straus, 
Rothenberg,  &  Daugherty,  2009). 

Topic  and  Content  Selection 

After  reviewing  existing  IMI,  two  topics  were  selected  on  which  to  design  six  IMI 
exemplars:  Adjust  Indirect  Fire  (Task  #061-283-6003)  and  Conduct  a  Defense  by  a  Squad  (Task 
#071-430-0002)  (U.S.  Army,  2008).  Given  the  emphasis  of  the  MCoE  initiative  Squad  as  the 
Foundation  of  the  Decisive  Force ,  the  two  topic  domains  were  selected  because  of  their 
relevance  to  new  squad  and  team  leaders  in  combat  arms  specialties  (Maneuver  Center  of 
Excellence,  n.d.).  Table  2  describes  the  six  IMI  exemplars  in  relation  to  the  two  topic  domains. 

Given  that  the  target  audience  was  new  squad  or  team  leaders  (i.e.,  Specialist/Corporals, 
E-4,  and  Sergeants,  E-5),  it  was  anticipated  that  Conduct  a  Defense  by  a  Squad  would  be  more 
familiar  to  Soldiers  because  it  is  a  common  task  in  which  Soldiers  at  varying  levels  participate, 
even  if  they  are  not  yet  in  the  role  of  decision-makers  or  leaders.  In  contrast,  it  was  anticipated 
that  Adjust  Indirect  Fire  would  be  less  familiar  to  Soldiers  because  it  is  a  more  complex  topic, 
with  specialized  training  requirements. 
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Table  2 


Six  Point  of  Need  IMI  Exemplars  by  Topic,  Type  ofIMI,  and  Title 


Topic 

Type  of  IMI 

Title 

Adjust  Indirect  Fire 
(Task  #061-283-6003) 

Familiarization 

Engaging  Targets  with  Supporting  Fires 

Core 

Conduct  Immediate  Suppression 

Tailored  Training 

Adjust  Indirect  Fire 

Conduct  a  Defense  by  a 
Squad 

(Task  #07 1-430-0002) 

Familiarization 

Prepare  Positions  for  Crew-Served  Weapons 

During  an  Urban  Operation 

Core 

Designate  and  Prepare  Urban  Fighting  Positions 
for  a  Javelin  Team 

Tailored  Training 

Conduct  a  Defense  by  a  Squad  in  an  Urban 
Operation 

Note:  The  tasks,  conditions,  and  standards  for  the  selected  topics  are  documented  in  the  Soldier’s  Manual 
of  Common  Tasks,  Warrior  Leader  Skills  Level  2,  3,  and  4  (U.S.  Army,  2008). 


Three  Variations  of  Needs  Focused  IMI 

The  three  types  of  training — familiarization,  core/refresher,  and  tailored  training — were 
selected  to  cover  a  variety  of  training  needs  from  preparation  for  a  course,  acquiring  background 
knowledge  to  support  more  in-depth  study,  supporting  execution  of  tasks  on  the  job,  etc. 
Familiarization  and  core/refresher  training  were  not  tailored,  whereas  the  tailored  training 
condition  was  designed  to  facilitate  learners  in  selecting  an  individualized  learning  path. 

Familiarization.  Familiarization  training  was  intended  to  provide  learners  with  an 
overview  of  the  topic  and  resources  to  learn  more  (i.e.,  breadth  of  information).  It  would  assist 
learners  in  gaining  a  modest  understanding  of  the  topic  domain,  but  not  a  high-level  of 
proficiency  or  expertise.  Familiarization  IMI  was  further  intended  for  learners  who  need  to  be 
prepared  for  subsequent  and  more  in  depth  learning,  assisting  them  in  forming  a  basic 
understanding  of  the  content  domain.  Based  on  a  previous  survey  of  Army  Combat  Arms  IMI,  it 
was  found  that  familiarization  is  the  most  common  design  used  (Blankenbeckler  et  al.,  2013). 

Core/Refresher.  Core/refresher  training  was  intended  to  provide  only  the  essential 
information  learners  would  need  to  perform  a  defined  task  (i.e.,  it  provided  depth  of  information, 
with  less  breadth).  When  designing  core/refresher  IMI,  the  model  was  ‘how-to’  manuals  that 
present  step-by-step  procedures  of  how  to  execute  a  task.  Again,  given  that  the  focus  was  on 
learners’  immediate  needs,  they  were  not  intended  to  reach  a  high-level  of  proficiency. 
Core/refresher  IMI  was  designed  for  learners  who  would  need  immediate  support  in  conducting 
on-the-job  tasks  or  a  quick  refresher  of  a  previously  learned  skill.  This  type  of  training  was  not 
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intended  to  lead  to  mastery  of  the  task,  as  the  core/refresher  IMI  contained  no  hands-on 
performance  or  sophisticated  simulation.  This  variation  was  selected  based  on  discussion  with 
stakeholders  at  MCoE  DOTD,  who  expressed  a  need  to  explore  IMI  designs  that  would  support 
rapid,  in-depth  refresher  training. 

Tailored  Training.  Tailored  training  combined  both  breadth  and  depth  of  information 
into  multiple  user-selected  learning  paths  while  also  providing  increased  learner  autonomy  over 
the  familiarization  and  core/refresher  variations.  It  also  included  pre-  and  post-training 
diagnostic  assessments  with  individualized  feedback  following  each  assessment.  Feedback  to 
learners  was  based  on  a  pre-training  diagnostic  assessment.  The  feedback  was  intended  to  help 
learners  identify  content  areas  in  which  they  were  deficient  and  to  select  among  the  available 
learning  paths.  After  learners  completed  training,  they  were  again  administered  a  diagnostic 
assessment  and  received  feedback,  giving  them  the  option  for  additional  training.  The  tailored 
training  design  was  intended  to  meet  a  variety  of  learning  needs.  The  diagnostic  assessments 
were  intended  to  help  learners  to  be  more  aware  of  their  specific  learning  needs  and  to  make 
reasonable  decisions  on  how  to  tailor  their  learning  experience.  While  this  approach  to  tailoring 
did  not  make  use  of  cutting-edge  technologies  (such  as  computer-adaptive  testing),  it  represented 
an  effective  and  low-cost  solution  to  enable  tailored  training  in  Army  IMI.  This  IMI  design  was 
selected  to  support  objectives  described  in  the  Army  Learning  Concept,  which  focus  on 
developing  and  delivering  training  that  can  address  Soldiers’  specific  learning  needs  (TRADOC, 
2011). 

Design  Principles  Applied  in  the  IMI 

In  designing  the  IMI  exemplars  for  the  experiment,  configurations  of  design  principles 
were  applied.  One  design  feature  shared  across  the  different  types  of  IMI  was  content  chunking. 
Content  chunking  is  a  design  strategy  that  breaks  up  content  into  smaller  parts  that  can  be  more 
readily  learned,  related  to  Miller’s  (1956)  research  on  the  limitations  of  human  short-term 
memory.  We  chunked  content  in  terms  of  whole  tasks  and  parts  of  tasks  (see  Figure  1). 

Content  chunking  allowed  the  researchers  to  present  an  organized  structure  to  the  learner 
up-front,  as  well  as  consolidate  closely  related  material  into  manageable  parts,  helping  to  manage 
cognitive  load  (Sweller,  2005)  and  reduce  extraneous  processing  (Mayer,  2005).  In  the  example 
presented  in  Figure  1,  content  is  being  initially  sorted  into  two  general  categories — i.e.,  Call  for 
Fire,  and  Subsequent  Corrections.  Within  each  of  these  sections,  content  was  further  chunked  so 
that  training  content  would  be  presented  in  short,  easily  completed  sections.  This  chunking 
structure  was  intended  to  facilitate  learners  in  navigating  through  the  IMI,  serving  as  a  way  to 
organize  content  for  the  less  knowledgeable  Soldiers  or  as  a  road-map  for  the  more 
knowledgeable.  While  chunked  content  was  not  intended  to  increase  learner  autonomy,  the 
section  headings  could  be  used  by  some  learners  to  navigate  around  portions  of  content. 
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X* 


Complete  each  of  the  topics  in  sequence 


Call  for  Fire 


Subsequent  Corrections 


Learning  Objective 

Upon  successful  completion  of  this  lesson  you  will  be  familiar 
with  the  elements  and  composition  of  a  call  for  fire. 

An  accurate  and  timely  call  for  fire  will  enable  the  small  unit 
leader  to  Engage  Targets  with  Supporting  Fires,  when  required. 


Page  3  of  6 


00 


Figure  1 .  Example  of  Content  Chunking 

Making  sure  the  instruction  had  a  narrative  pattern,  presenting  a  clear  sequence  of  steps 
to  interconnect  parts  of  the  tasks,  was  also  an  important  design  principle  (Clark,  2005).  This 
approach  was  particularly  useful  for  the  core/refresher  variations  of  needs-focused  IMI.  For 
example,  questions  and  feedback  followed  the  sequence  of  an  Adjust  Indirect  Fire  mission  to 
summarize  the  Call  for  Fire  and  Fire  Direction  Center  (FDC)  responses  (see  Figure  2). 
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Figure  2.  Example  of  Narrative  Flow 

Evaluation  and  feedback  on  specific  task  steps  was  also  expected  to  help  guide  learners 
through  the  learning  process,  and  served  as  a  particular  way  of  implementing  narrative 
sequencing  (see  Figure  3).  This  approach  helped  learners  to  test  their  knowledge  throughout  the 
training  and  to  maintain  their  focus  on  relevant  content.  This  was  viewed  as  an  alternative  way 
to  approach  ‘checks-on-leaming’1  and  was  intended  to  maintain  focus  on  the  sequence  of  task 


1  Many  examples  of  existing  Army  IMI  used  a  checks-on-leaming  design  pattern.  They  would  present  three  to  four 
pages  of  learning  content,  then  present  questions  on  that  brief  section  of  content.  Often,  the  feedback  provided 
concerned  whether  the  learner  had  answered  correctly  or  incorrectly. 
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execution.  The  learner  received  and  responded  to  questions  for  each  step  in  the  task  as  it  built  in 
complexity.  They  were  presented  feedback  on  their  decisions  between  steps,  building  toward  the 
complete  task  (van  Merrienboer  &  Kester,  2005). 


Second  Call  for  Fire  Transmission 


NK  415128,  Over. 


From  my  location  at  NK  415114, 
direction  360°,  distance  1,400,  Over. 


T'our  Call  Sign:  W  22 
UC  Call  Sign:  F  73 
Iwiated  location  of 
the^my:  NK  415128 


Grid,  NK415128,  direction  0°,  Over. 


Second  Call  for  Fire  Transmission:  Correct  Feedback 


Second  Call  for  Fire  Transmission:  Incorrect  Feedback 


INCORRECT! 

Your  second  transmission  to 
the  FDC  must  provide  the 
target  location  using  the  grid, 


Your  Call  Sign:  W  22 
FDC  Call  Sign:  F  73 
Estimated  location  of 


the  enemy:  NK  4151 28 


Tailored  Training: 
Adjust  Indirect  Fire 


The  FDC  responds,  “This  is 
F  73,  Adjust  Fire,  Out" 
Which  message,  below 
provides  the  correct  second 
transmission  for  the  call  for 
fire  in  this  situation? 


CORRECT! 

Your  second  transmission  to 
the  FDC  provides  the  target 
location  using  the  grid,  polar. 
or  shift  from  a  known  point 
method.  The  grid  method  was 
conveyed  in  the  warning 
order.  When  the  grid  method 
is  used,  the  coordinates 
should  locate  the  target  to  the 
nearest  100  meters  (six  digits) 
and  include  the  100,000-meter 
grid  square  identification.  Your 
selection  was  correct:  "NK 
415128,  Over." 


00 


oo 


Tailored  Training: 
Adjust  Indirect  Fire 


Figure  3.  Example  of  Questions  and  Feedback  on  Individual  Task  Steps 

Finally,  the  tailored  training  variation  of  needs-based  IMI  had  some  of  its  own  unique 
features  to  support  increased  learner  autonomy.  For  instance,  whole  task  evaluation  and 
feedback  was  unique  to  this  variation.  The  tailored  training  IMI  was  designed  with  a  diagnostic 
assessment  at  the  start.  This  pre-training  diagnostic  was  designed  to  measure  prior  knowledge 
learners  had  concerning  the  content  domain  and  provided  feedback  to  learners  on  their 
deficiencies.  It  was  structured  such  that  learners  would  receive  individualized  feedback  and 
recommendations  for  particular  content  areas  that  they  should  focus  on  in  training.  Feamers 
were  allowed  to  select  the  content  they  wanted  leam,  although  the  recommendations  were 
intended  to  help  them  to  optimize  their  self-selected  learning  path.  Training  recommendations 
were  made  in  relation  to  the  structure  of  chunked  content  within  the  IMI,  enabling  learners  to 
identify  how  to  navigate  through  the  content.  This  simple  approach  enabled  the  researchers  to 
adapt  training  to  learners’  needs  without  employing  a  complex  learning  management  system,  as 
the  learner  ultimately  determined  how  they  would  proceed  through  the  IMI.  Once  the  IMI  was 
complete,  learners  were  again  given  an  opportunity  to  test  their  knowledge  and  were  provided 
with  additional  feedback  to  help  guide  their  selection  of  future  learning  (see  Figure  4). 
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1.  Designate/prepare/adjust 
primary  individual/team/ 
crew  fighting  positions  to: 

Provide  effective/ 
adequate  fields  of  fire. 

£1 

Enhance  Soldier/unit 
survival  and  safety. 

l3 

2.  Designate/prepare: 

-  Alternate  positions 

-  Supplementary 
positions 

(These  positions  should 
meet  the  same  standards 
as  primary  positions.) 

^1 

12] 

3.  Employ  fire  control 

l3 

measures  effectively 

3 

3 

If  all  categories  are  a  GO,  congratulations,  you 
have  attained  the  standards  for  this  lesson  and 
may  end  training  or  elect  to  take  selected  topics. 
If  you  elect  to  end  training,  select  NEXT  to  EXIT. 

If  any  elements  of  the  task  was  assessed  as  a 
NO  GO,  the  Recommended  Training  Topic  or 
Topics  have  been  illuminated.  Upon  completion 
of  additional  training  you  will  be  routed  to  the 
Post-Training  Assessment. 

Suggested  sections  you  should  review: 

•  Platoon  and  Squad  Defensive  Planning 

-  Selecting  Weapon  Positions  and 
Controlling  Fires 


Assessment  Scoring:  =  GO 


=  NO  GO 


Tailored  Training:  Conduct  a  Defens 
by  a  Squad  in  an  Urban  Operation 


Recommended  Training  Topics 


Select  a  topic  to  continue  training. 


Platoon  and  Squad  Defensive  Planning 


Selecting  Weapon  Positions  and 
Controlling  Fires 


Common  Preparation  Considerations 


Prepare  Machine  Gun  Positions 


Prepare  Antitank  Weapon  Positions 


Page  11  of  13 


Figure  4.  Example  of  Whole-Task  Diagnostic  Assessment  and  Feedback  in  Tailored  Training 
Condition 

Experimental  Evaluation  of  the  IMI 

The  IMI  were  experimentally  tested  with  91  Soldiers  enrolled  in  the  Warrior  Leader 
Course  (WLC)  at  the  Henry  Caro  Noncommissioned  Officer  Academy  at  Fort  Benning,  GA. 
The  experiment  was  designed  to  test  the  three  variations  of  needs-focused  IMI  across  the  two 
topic  domains.  One  domain  was  more  familiar  (i.e.,  Conduct  a  Defense  by  a  Squad)  to  Soldiers 
and  the  other  was  less  familiar  (i.e.,  Adjust  Indirect  Fire).  The  three  types  of  IMI  tested  were 
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familiarization,  core/refresher,  and  tailored  training.  Familiarization  and  core/refresher  were 
non-tailored  IMI  variations,  whereas  the  tailored  training  IMI  incorporated  design  features  to 
address  individual  learning  needs. 

The  IMI  were  tested  in  six  two-hour  sessions.  All  three  IMI  variations  were  tested  in 
each  session  with  between  15  to  19  Soldiers  enrolled  in  WLC.  No  Soldiers  participated  in  more 
than  one  session.  At  the  start  of  a  session,  the  Soldiers  were  briefed  on  the  purpose  of  the 
research  and  administered  and  signed  the  informed  consent  form.  They  were  then  asked  to 
complete  a  short  demographic  questionnaire  and  to  rate  their  familiarity  with  the  selected 
training  topic  for  that  session,  described  broadly  as  either  Adjust  Indirect  Fire  or  Conduct  a 
Defense  by  a  Squad.  Based  on  their  ratings,  Soldiers  were  assigned  to  one  of  the  three  IMI 
conditions:  familiarization  ( n  =  29;  32%  of  Soldiers),  core/refresher  (n  =  29;  32%),  or  tailored 
training  in  =  33;  36%).  A  self-rating  assessment  of  prior  knowledge  about  the  topic  was  used  to 
balance  assignment  of  Soldiers.  Based  on  self-ratings  of  prior  knowledge,  Soldiers  who  rated 
themselves  1  to  3,  4  to  6,  or  7  to  9  on  the  prior  knowledge  scale  were  sorted  respectively  into 
three  groups.  From  these  three  groups,  Soldiers  were  assigned  to  a  specific  IMI  variation, 
balancing  prior  knowledge  across  each  variation.  Soldiers  who  participated  were 
Specialists/Corporals  (E-4,  n  =  61;  67%)  and  Sergeants  (E-5,  n  =  30;  33%),  and  most  had 
deployment  experience  ( n  =  68;  75%).  Some  had  hands-on  experience  with  the  relevant  topic 
domains,  with  fewer  having  experience  with  Adjust  Indirect  Fire  (n  =  15;  16%)  than  with 
Conduct  a  Defense  by  a  Squad  ( n  =  35;  38%).  Both  Combat  Arms  (n  =  28;  31%)  and  other 
Miltiary  Occupational  Specialties  (MOSs)  (n  =  63;  69%)  were  represented. 

Once  assigned  to  their  IMI  variation,  each  Soldier  was  administered  a  pre-training  test  for 
their  appropriate  topic.  Each  topic  had  two  alternate  test  forms  that  were  matched  question-to- 
question  for  content  and  difficulty.  If  a  Soldier  received  Form  A  before  training,  he  or  she 
received  Form  B  after  training,  and  vice  versa.  Once  the  pre-test  was  complete,  it  was  turned  in 
to  a  data  collector,  the  time  was  recorded,  and  Soldiers  were  allowed  to  begin  training  at  their 
own  pace  with  the  IMI.  At  the  completion  of  training,  Soldiers  completed  a  posttest.  Finally, 
Soldiers  were  asked  to  fill  out  a  user  experience  questionnaire;  approximately  15  of  the  Soldiers 
were  briefly  interviewed  about  their  learning  experiences  with  the  IMI  following  the  session. 

The  complete  session  required  approximately  2  hours  to  complete. 

RESULTS 

Given  that  the  tailored  training  variation  was  designed  to  provide  learners  the  greatest 
flexibility  to  address  their  learning  needs,  the  researchers  expected  the  largest  increase  in  pretest 
versus  posttest  scores  for  the  two  tailored  training  groups  compared  to  the  non-tailored 
familiarization  and  core/refresher  IMI  variations.  Also,  the  researchers  expected  that  learners  in 
the  tailored  training  group  would  have  higher  point  gains  from  pretest  to  posttest  for  the  less 
familiar  content  domain  (Adjust  Indirect  Fire)  compared  to  the  more  familiar  content  domain 
(Conduct  a  Defense  by  a  Squad). 
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Prior  Knowledge  and  Experience 


To  verify  the  assumption  that  the  WLC  Soldiers  had  different  background  knowledge  and 
experience  with  the  respective  topic  domains,  their  prior  knowledge  ratings  were  evaluated  (see 
Table  3).  Conduct  a  Defense  by  a  Squad  (M  =  5.04,  ‘moderate’  knowledge;  SD  =  2.19)  rated 
higher  than  Adjust  Indirect  Fire  ( M  =  3.30,  ‘none  to  little’  knowledge;  SD  =  2.15),  F(  1 , 89)  = 
14.73,  p  <  0.001.  This  finding  supported  the  initial  assumption  concerning  Soldiers’  levels  of 
familiarity  with  the  two  content  domains.  In  addition,  more  Soldiers  indicated  having  prior 
experience  with  the  Conduct  a  Defense  by  a  Squad  domain  (n  =  35;  38%)  compared  to  Adjust 
Indirect  Fire  (n  =  15;  16%),X2(1,  N=  91)  =  5.12,  p  =  0.017. 

Table  3 

Soldiers  ’  Ratings  of  Prior  Knowledge  by  Training  Topic 

and  Type  of  Needs-Focused  IMI  Training _ 

Adjust  Indirect  Fire  Conduct  a  Defense  by  a 

Squad 

Mean  Mean 


(SD) _ (SD) 


Fam 

Core 

Tailored 

Fam 

Core 

Tailored 

2.93 

(1.86) 

3.43 

(2.24) 

3.50 

(2.39) 

5.27 

(2.15) 

5.33 

(2.09) 

4.6 

(2.35) 

*Note:  The  self-ratings  of  prior  knowledge  were  scaled  as  T  to  3  ’  = 
‘None  to  Little’,  ‘4  to  6’  =  ‘Moderate’,  and  ‘7  to  9’  =  ‘Extensive’. 


Reliability  of  Tests 

Individual  differences  in  background  knowledge  and  experience  may  also  have  been 
reflected  in  the  reliability  metrics  for  both  versions  of  the  pretest  and  posttests  administered. 
Table  4  presents  the  reliabilities  for  test  versions  A  and  B,  administered  as  either  pretests  or 
posttests  for  Adjust  Indirect  Fire  and  Conduct  a  Defense  by  a  Squad. 

Table  4 


Reliabilities  for  Pretests  and  Posttests 


Test  Version 

Adjust  Indirect  Fire 

Conduct  a  Defense  by  a  Squad 

Pretest 

Posttest 

Pretest 

Posttest 

A 

0.63 

0.81 

0.84 

0.81 

B 

0.79 

0.87 

0.84 

0.78 

Note:  Unbiased  parallel  forms  reliabilities  are  reported. 


Reliability  measures  for  the  alternate  tests  were  in  the  good  to  excellent  range  overall, 
with  the  exception  of  Adjust  Indirect  Fire  at  pretest.  At  posttest,  the  reliabilities  the  Adjust 
Indirect  Fire  tests  were  in  the  good  to  excellent  range.  It  is  likely  that,  given  Soldiers’  lack  of 
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familiarity  with  the  Adjust  Indirect  Fire  topic,  the  lower  reliabilities  at  pretest  may  reflect 
increased  guessing. 

Training  Time 

The  IMI  were  designed  to  take  roughly  an  equivalent  time  to  complete,  with  an  exception 
for  the  diagnostic  testing  in  the  tailored  training  condition.  The  amount  of  time  that  Soldiers 
spent  on  the  training  was  roughly  equivalent  ( M  =  12  to  16  minutes)  across  all  conditions,  after 
subtracting  out  the  additional  time  for  the  diagnostic  assessment  in  the  tailored  training  condition 
(M  =  15  to  18  minutes).  The  exception  to  this  finding  was  the  significantly  higher  time  to 
complete  core/refresher  training  ( M  =  23  minutes)  in  the  Conduct  a  Defense  by  a  Squad 
condition,  /? standardized  =  0.35,  t  =  2.49,  p  =  0.016,  partial  r  =  0.35.  This  core/refresher  IMI  was 
focused  on  preparing  urban  fighting  positions  for  the  Javelin  missile  team. 

Comparison  of  IMI  Variations  and  Topic  Domains 

A  repeated  measures  analysis  of  variance  (ANOVA)  was  used  to  evaluate  the  relationship 
between  pretest  and  posttest  scores  for  the  three  variations  of  needs  focused  IMI,  and  the  two 
topic  domains.  Prior  knowledge,  deployments,  prior  experience,  and  MOS  (i.e.,  Combat  Arms 
vs.  Non-Combat  Arms)  were  included  as  covariates.  There  was  an  overall  significant  within- 
subjects  increase  in  scores  between  pretest  and  posttest  for  both  Adjust  Indirect  Fire  and  Conduct 
a  Defense  by  a  Squad  across  the  three  IMI  variations,  F(  1,  81)  =  14.67,  p  <  0.001,  l~lP2  =  0.15. 2 
There  was  a  mean  12%  point  increase  from  pretest  to  posttest  for  Adjust  Indirect  Fire,  and  a 
mean  10.7%  point  increase  for  Conduct  a  Defense  by  a  Squad.  Table  5  presents  the  mean  pretest 
and  posttest  scores  by  topic  domain  and  IMI  type. 

Table  5 

Soldiers  ’  Pretest  and  Posttest  Mean  Percentage  Correct  by  Training  Topic  and  Type  of  Needs - 
Focused  IMI  Training 

Measures  Adjust  Indirect  Fire  Conduct  a  Defense 

by  a  Squad 

Mean  Mean 


(SD) 

(SD) 

Fam 

Core 

Tailored 

Fam 

Core 

Tailored 

Pretest  Score  (Mean  %  Items  Correct) 

39% 

(13%) 

34% 

(16%) 

48% 

(11%) 

51% 

(13%) 

51% 

(12%) 

45% 

(15%) 

Posttest  (Mean  %  Items  Correct) 

47% 

(18%) 

44% 

(21%) 

66% 

(17%) 

60% 

(12%) 

63% 

(11%) 

56% 

(13%) 

There  were  three  between-subjects  effects  identified.  Soldiers  who  were  in  Combat 
Arms  MOSs  tended  to  have  greater  point  gains  compared  to  Soldiers  who  were  not  in  Combat 
Arms  MOSs,  F(l,  81)  =  16.71,  p  <  0.001,  F)p2  =  0.15.  Soldiers’  test  scores  for  the  more  familiar 
domain  (Conduct  a  Defense  by  a  Squad)  were  higher  overall  (M increase  =  7%  points)  compared  to 


2  tjp2  is  partial  eta-squared,  a  measure  of  effect  size. 
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the  test  scores  for  the  less  familiar  domain  (Adjust  Indirect  Fire),  F(l,  81)  =  7.24, 
=  0.08.  Finally,  there  was  an  interaction  of  IMI  type  by  topic  domain,  F( 2,  81)  = 
l~lP2  =  0.16.  Table  6  summarizes  the  specific  interaction  effects. 

Table  6 

Summary  of  Interactions  for  the  Repeated  Measures  AN  OVA 

p  =  o.oo9,  rii>2 
7.43, =  0.001, 

Pre/Post  Topic 

Variable 

P 

t 

P 

Hp 

Test 

Pretest  Adjust  Indirect 

Intercept 

45.8 

9.44 

0.000 

0.707 

Fire 

Prior  Knowledge  (Self-Rating) 

0.35 

0.31 

0.760 

0.003 

Deployment 

-7.31 

-1.74 

0.090 

0.076 

Prior  Experience 

-8.72 

-1.45 

0.155 

0.054 

Combat  Arms  MOS 

19.03 

3.32 

0.002 

0.229 

Familiarization 

-4.07 

-0.89 

0.382 

0.021 

Core/Refresher 

-9.60 

-2.10 

0.043 

0.107 

Tailored  Training 

— 

— 

— 

Conduct  a  Defense 

Intercept 

37.05 

6.65 

0.000 

0.525 

by  a  Squad 

Prior  Knowledge  (Self-Rating) 

0.59 

0.58 

0.566 

0.008 

Deployment 

1.88 

0.38 

0.703 

0.004 

Prior  Experience 

0.30 

0.07 

0.943 

0.000 

Combat  Arms  MOS 

12.69 

2.79 

0.008 

0.163 

Familiarization 

2.53 

0.58 

0.568 

0.008 

Core/Refresher 

5.32 

1.18 

0.245 

0.034 

Tailored  Training 

" 

— 

— 

— 

Posttest  Adjust  Indirect 

Intercept 

69.72 

11.66 

0.000 

0.786 

Fire 

Prior  Knowledge  (Self-Rating) 

-1.27 

-0.90 

0.376 

0.021 

Deployment 

-14.87 

-2.88 

0.007 

0.183 

Prior  Experience 

-12.21 

-1.65 

0.108 

0.068 

Combat  Arms  MOS 

32.71 

4.63 

0.000 

0.366 

Familiarization 

-11.54 

-2.04 

0.049 

0.101 

Core/Refresher 

-15.24 

-2.71 

0.010 

0.165 

Tailored  Training 

— 

— 

— 

— 

Conduct  a  Defense 

Intercept 

49.95 

8.83 

0.00 

0.661 

by  a  Squad 

Prior  Knowledge  (Self-Rating) 

0.64 

0.62 

0.54 

0.010 

Deployment 

4.91 

0.99 

0.33 

0.024 

Prior  Experience 

-3.41 

-0.81 

0.43 

0.016 

Combat  Arms  MOS 

4.64 

1.00 

0.32 

0.025 

Familiarization 

2.54 

0.57 

0.57 

0.008 

Core/Refresher 

4.97 

1.09 

0.28 

0.029 

Tailored  Training 

— 

— 

— 

— 

Note:  Statistically  significant  interactions  are  in  bold. 
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A  difference  was  found  among  the  three  types  of  needs-focused  IMI  for  the  less  familiar 
content  domain  (Adjust  Indirect  Fire).  For  the  less  familiar  domain,  tailored  training  was 
associated  with  higher  posttest  scores  (i.e.,  8  to  10%  points),  over  both  familiarization  and 
core/refresher  training.  As  a  caveat,  core/refresher  scores  for  Adjust  Indirect  Fire  were  lower 
than  tailored  training  at  the  pretest. 

Finally,  the  effects  of  whether  Soldiers  had  deployed  or  were  in  Combat  Arms  MOSs 
were  associated  with  their  posttest  scores.  Having  at  least  one  deployment  was  associated  with 
lower  posttest  scores  (a  reduction  of  approximately  12%  points),  whereas  being  in  a  Combat 
Arms  MOS  was  associated  with  higher  pretest  (an  increase  of  approximately  16%  points), 
posttest  scores  (an  increase  of  approximately  15%  points). 

There  was  no  effect  for  training  type  for  Conduct  a  Defense  by  a  Squad,  nor  were  self- 
ratings  of  prior  knowledge  associated  with  posttest  scores  for  either  content  domain.  Figure  5 
summarizes  the  key  results  for  topic  domain  and  type  of  IMI  design. 
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Adjust  Indirect  Fire: 
Tailored  point  gain  (18  pts) 
differed  significantly  from 
core  (10  pts)  and 
familiarization  (8  pts). 

Overall  point  gain  was  also 
statistically  significant. 


Conduct  a  Defense 
by  a  Squad: 

No  significant 
difference  in  gains 
between  IMI 
variations. 

Overall  point  gain 
was  statistically 
significant. 


■  Pretest 

■  Posttest 

Note: These  were 
paper-base  dte  sts. 


Fam  Core  Tailored 

Adjust  Indirect  Fire 


Fam  Core  Tailored 

Conduct  a  Defense 


Figure  5.  Pretest  and  Posttest  Scores  by  Content  Domain  and  Training  Type 

Due  to  small  sample  size,  analyses  were  limited  at  the  level  of  the  tailored  training  IMI 
variations.  Even  so,  we  explored  whether  there  were  relationships  between  the  diagnostic  pre¬ 
assessment  feedback  Soldiers  received  and  their  selection  of  instructional  content  in  the  tailored 
training  IMI  condition.  When  looking  only  at  the  instructional  path  Soldiers  selected,  there  did 
not  appear  to  be  a  definitive  relationship.  However,  when  correlating  training  time  within  each 
learning  path  of  the  Adjust  Indirect  Fire  IMI,  there  were  statistical  trends  which  suggested  that 
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Soldiers  may  have  spent  more  time  on  some  sections  of  content  (i.e.,  slowed  down  or  paid  more 
attention)  given  their  individualized  feedback  they  had  received.  For  instance,  Soldiers  with 
lower  diagnostic  pre-assessment  scores  for  ‘Formulate  a  Call  for  Fire’  were  more  likely  to  spend 
greater  time  in  the  ‘Fundamental  Call  for  Fire  Training’  section  of  the  IMI,  r(l  1)  =  -0.61,  p  = 
0.048.  In  addition,  there  were  marginally  significant  trends  indicating  a  similar  pattern  for  the 
other  sections  of  the  IMI:  time  spent  on  ‘Comprehensive  Call  for  Fire  and  Adjust  Indirect  Fire 
Training’  trended  with  the  ‘Determine/Transmit  Elements  of  a  Call  for  Fire’  scores,  r(16)  =  - 
0.48,  p  =  0.06;  and  time  spent  on  ‘Fundamental  Adjustment  Training’  was  associated  with 
diagnostic  pre-assessment  ‘Adjust  Rounds  on  Target;  End  the  Mission’  scores,  r(12)  =  -0.52,  p  = 
0.081.  With  a  larger  sample  size,  a  clearer  picture  would  likely  emerge  of  how  Soldiers  were 
using  the  diagnostic  assessment  feedback  they  received  in  the  Adjust  Indirect  Fire  IMI  condition. 
Although  the  overall  statistical  result  for  the  tailored  training  variation  of  Adjust  Indirect  Fire 
was  positive,  to  determine  how  diagnostic  feedback  may  have  influenced  Soldiers  to  attend  more 
carefully  to  recommended  content  would  require  additional  data. 

The  associations  between  training  time  and  diagnostic  feedback  for  the  Conduct  a 
Defense  by  a  Squad  group  were  less  apparent.  For  the  section  of  diagnostic  assessment  labeled 
‘Provide  effective/adequate  fields  of  fire,’  higher  scores  were  associated  with  more  time  spent  on 
the  ‘Selecting  Weapons  Positions  and  Controlling  Fires’  section  of  the  IMI,  r(17)  =  0.46,  p  = 
0.032.  A  similar  correlation  was  noted  between  the  ‘Employ  fire  control  measures  effectively’ 
section  of  the  diagnostic  assessment  and  the  ‘Selecting  Weapons  Positions  and  Controlling  Fires’ 
section  of  the  IMI,  r(17)  =  0.43,  p  =  0.044.  There  were  no  marginal  trends  noted  for  the  other 
variables,  associating  training  time  with  diagnostic  pre-assessment  feedback.  The  diagnostic 
feedback  provided  to  Soldiers  did  not  appear  to  be  as  influential  in  the  Conduct  a  Defense  by  a 
Squad  (more  familiar)  IMI  condition  as  it  was  in  the  Adjust  Indirect  Fire  (less  familiar)  IMI 
condition. 

Learner  Experiences  and  Feedback 

IMI  is  effective  if  learners  perceive  value  in  using  it  to  learn.  In  order  to  explore 
Soldiers’  experiences  using  the  IMI,  the  researchers  developed  and  administered  a  user 
experience  questionnaire.  This  questionnaire  was  administered  after  all  IMI  training  and  testing 
was  complete. 

Initially,  39  items  were  included  on  the  questionnaire.  An  exploratory  factor  analysis 
was  conducted,  reducing  the  39  items  to  31  items  in  six  factors,  accounting  for  58%  of  variance.3 
Factors  were  selected  if  they  contributed  at  least  3%  to  the  explained  variance.  While  the  sample 
size  was  small  (N  =  91),  the  tests  for  sampling  adequacy  and  correlation  were  good:  Kaiser- 
Meyer-Olkin  =  0.84;  Bartlett’s  Test  X2(df=  630)  =  2,339.023,  p  <  0.001.  Item  level  means  and 
standard  deviations  are  reported  in  Appendix  A. 


3  The  factor  analysis  extraction  method  was  Alpha  factoring  with  Promax  (Kappa  =  4)  rotation.  Alpha  factoring 
maximizes  the  reliability  coefficient  of  items  to  combine  them  into  factors.  Promax  rotation  is  an  orthogonal 
rotation  that  is  less  strict  than  Varimax,  allowing  for  oblique  adjustments  following  an  initial  orthogonal  solution 
(Kline,  2013). 
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Factor  scores  were  calculated.  Table  7  summarizes  the  results  for  each  factor  measured 
on  the  user  experience  questionnaire  for  each  IMI  variation  by  content  type.  The  questionnaire 
used  Likert  style  response  options,  ‘Strongly  Disagree’  (1)  to  ‘Strongly  Agree’  (5). 


Table  7 


User  Experience  Factor  Scores  by  Needs-Based  IMI  Variation  and  Type  of  Content. 


User  Experience  Factors  and  Definitions 

User  Experience  Scores 

Mean 

(SD) 

Adjust  Indirect  Fire 

Conduct  a  Defense 
by  a  Squad 

Fam 

77=14 

Core 
77=  14 

Tailored 

77=16 

Fam 

77=  1 5 

Core 

77=15 

Tailored 

77=17 

Quality  of  Learning  Experience  (a  =  0.91) 

•  Learners’  overall  view  of  their  learning 

3.79 

3.55 

3.79 

4.03 

3.92 

3.61 

experience  with  the  IMI. 

(0.75) 

(0.82) 

(0.65) 

(0.81) 

(0.62) 

(0.77) 

Quality  of  Design  and  Content  (a  =  0.86) 

•  Learners’  overall  impression  of  the  IMI 

4.19 

3.96 

3.98 

4.23 

4.08 

4.02 

design  and  content. 

(0.46) 

(0.54) 

(0.56) 

(0.65) 

(0.50) 

(0.50) 

Continuity  of  Topics  (a  =  0.85) 

•  Learners’  perceived  connections  between 

4.14 

3.93 

4.03 

4.33 

4.15 

3.95 

topics  in  the  IMI. 

(0.47) 

(0.57) 

(0.48) 

(0.57) 

(0.55) 

(0.52) 

Credible  Examples  (a  =  0.85) 

•  The  examples  provided  in  the  IMI  were 

4.04 

3.88 

4.07 

4.16 

4.10 

3.95 

realistic  and  lifelike. 

(0.54) 

(0.54) 

(0.58) 

(0.53) 

(0.49) 

(0.47) 

Focus  and  Relevance  (a  =  0.77) 

•  Checks  on  learning,  etc.,  were  perceived  to 

4.07 

3.83 

3.94 

4.27 

4.07 

4.12 

be  on  topic  and  relevant  to  the  learning  task. 

(0.49) 

(0.74) 

(0.73) 

(0.54) 

(0.58) 

(0.44) 

Tracking  Progress  (a  =  0.77) 

•  Learners  were  able  to  track  their  progress 

4.03 

3.64 

3.66 

3.77 

3.83 

3.65 

within  the  IMI  to  stop/resume  training. 

(0.75) 

(0.77) 

(0.91) 

(0.82) 

(0.88) 

(0.77) 

Given  the  small  sample  size  on  which  the  factor  analysis  was  based,  the  resulting  factor 
structure  may  not  be  optimal.  That  said,  the  factor  structure  may  be  practically  useful  for 
evaluating  user  experiences  of  IMI.  The  questionnaire  items  and  factor  structure  would  likely 
benefit  from  continued  refinement. 


The  user  experience  scores  were  compared  between  the  three  types  of  needs-focused  IMI 
and  the  two  types  of  content  domains  using  an  analysis  of  variance.  There  were  no  significant 
differences  found.  This  finding  suggests  that  user  experiences  were  similar  across  the  three  IMI 
variations  and  two  topic  domains. 

To  explore  the  relationships  between  pretest  and  posttest  scores  and  user  experiences,  a 
correlation  analysis  was  conducted.  Soldiers  who  tended  to  have  higher  scores  on  the  pretest 
also  had  higher  quality  of  learning  experience  ratings,  r(91)  =  0.23,  p  =  0.03,  suggesting  that 
prior  knowledge  was  being  used  to  evaluate  the  quality  of  the  learning  materials.  In  addition, 
Soldiers  with  Combat  Arms  MOSs  were  also  more  likely  to  have  higher  quality  of  learning 


26 


experiences,  r(91)  =  0.35,  p  =  0.001,  indicating  that  they  perceived  the  domain  content  to  be 
more  relevant.  Likewise,  Soldiers  with  higher  prior  knowledge  of  the  topics  also  rated  their 
learning  experiences  higher  than  those  who  did  not,  r(91)  =  0.34,  p  =  0.001.  Soldiers  with  higher 
prior  knowledge  also  perceived  greater  continuity  to  the  topics  covered  in  the  IMI,  r(91)  =  0.27, 
p  =  0.010,  and  greater  relevance  to  the  checks-on-leaming  and  other  assessments  used  within  the 
IMI,  r(91)  =  0.24,  p  =  0.024.  No  other  variables  were  significantly  correlated. 

DISCUSSION 

Soldiers’  learning  needs  vary  given  their  career  path,  career  point,  prior  knowledge,  and 
experiences.  These  differences  can  affect  how  well  Soldiers  learn  in  specific  educational  and 
training  contexts,  or  with  particular  subject  matter  and  media  (TRADOC,  201 1).  While  face-to- 
face  instruction  allows  an  instructor  and  student  to  address  specific  learning  needs,  the  IMI  used 
in  distributed  learning  situations  is  often  designed  to  reach  a  large  audience.  In  reaching  a  large 
audience,  it  becomes  necessary  to  include  more  generic  content  than  face-to-face  instruction. 
However,  there  are  design  features  that  can  be  incorporated  into  IMI  to  better  tailor  content  to 
the  learning  needs  of  specific  audiences  and  individual  learners.  This  research  was  intended  to 
address  how  individual  differences  between  learners  can  best  be  accounted  for  when  designing 
IMI  for  audiences  that  are  diverse  in  background  knowledge  and  experience. 

ARI  researchers  conducted  an  experiment  to  evaluate  three  different  needs-focused  IMI 
designs  in  relation  to  two  topic  domains,  one  familiar  to  our  audience  of  learners  and  one 
unfamiliar.  One  of  the  needs-focused  IMI  designs  incorporated  design  features  to  tailor  the 
training  to  individual  needs;  the  other  two  IMI  designs  were  not  tailored.  In  past  research,  Dyer, 
Singh,  and  Clark  (2005)  found  that  Soldiers  with  less  prior  knowledge  tended  to  benefit  from 
having  more  structured  training,  whereas  learners  who  had  more  prior  knowledge  benefitted 
from  being  able  to  selectively  adapt  training  to  their  own  needs.  This  current  research  designed 
IMI  to  provide  options  to  learners,  using  a  flexible  approach  to  both  guide  novice  learners  as 
well  as  provide  a  means  for  more  experienced  learners  to  navigate  to  content  relevant  to  them. 
The  tailored  training  variation  was  the  most  comprehensive  in  terms  of  the  content  and  structure 
provided  to  learners,  whereas  familiarization  IMI  focused  on  breadth  of  content  and 
refresher/core  IMI  focused  on  depth  of  content. 

It  was  hypothesized  that  tailored  training  IMI  would  be  associated  with  the  largest 
increase  in  scores  between  pretest  and  posttest  compared  to  the  non-tailored  IMI  variations, 
regardless  of  whether  the  topic  domain  was  more  or  less  familiar.  It  was  also  hypothesized  that 
Soldiers  in  the  less  familiar  topic  condition  would  benefit  more  from  tailored  training  than 
Soldiers  in  the  more  familiar  topic  condition. 

Based  on  the  results,  the  tailored  training  IMI  was  most  beneficial  to  learners  in  the  less 
familiar  content  domain,  but  it  did  not  appear  to  have  any  impact  over  non-tailored  variations  in 
the  more  familiar  content  domain.  This  finding  was  in  accord  with  Dyer  et  al.’s  (2005)  research. 
Among  the  many  potential  options  available  for  designing  tailored  training,  the  approach  used 
here  was  intended  to  be  a  simple  and  straightforward  way  to  incorporate  tailoring  techniques  into 
needs-focused  IMI.  The  design  allowed  for  a  higher  degree  of  learner  autonomy  than  other 
adaptive  training  designs  that  rely  on  complex  algorithms  to  select  and  sequence  training  for 
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learners.  Moreover,  many  of  the  design  features  that  supported  learner  autonomy — such  as 
chunked  content  to  support  flexible  navigation — were  incorporated  into  the  non-tailored 
variations.  For  more  knowledgeable  learners,  those  design  features  may  have  afforded  a  means 
to  tailor  their  learning  experience  even  in  the  non-tailored  IMI  variations. 

Experimental  Test  of  the  IMI  and  Topic  Variations 

In  the  experimental  test  of  the  needs-focused  IMI,  the  results  for  Adjust  Indirect  Fire, 
which  was  the  less  familiar  task,  were  as  expected.  In  the  Adjust  Indirect  Fire  groups,  all 
Soldiers  showed  improved  performance,  with  Soldiers  in  the  tailored  training  group  showing 
much  greater  improvement  over  those  in  the  non-tailored  familiarization  and  core/refresher  IMI 
groups.  However,  the  results  for  Conduct  a  Defense  by  a  Squad  did  not  follow  this  pattern.  For 
Conduct  a  Defense  by  a  Squad,  all  three  needs-focused  IMI  conditions  were  associated  with 
roughly  equivalent  improvement  in  test  scores. 

One  possible  explanation  for  this  difference  between  the  Adjust  Indirect  Fire  and 
Conduct  a  Defense  by  a  Squad  concerns  the  ways  in  which  the  Soldiers  were  navigating  through 
the  IMI.  What  seemed  to  be  happening  more  frequently  in  the  Conduct  a  Defense  by  a  Squad 
group  was  that  Soldiers  tailored  their  learning  experience  by  taking  advantage  of  the  design 
features  incorporated  for  navigation  into  all  the  IMI  conditions.  Based  on  the  researchers’ 
observations,  Soldiers  seemed  to  be  using  the  content  headings  and  their  own  prior  knowledge  of 
the  domain  to  track  down  novel  information  more  quickly,  navigating  around — or  quickly 
through — sections  of  content  that  were  already  familiar  to  them.  This  observation  was  supported 
anecdotally  by  feedback  from  participants  following  the  experiment. 

With  diagnostic  testing  and  feedback,  the  tailored  training  for  Adjust  Indirect  Fire  may 
have  provided  a  similar  support  to  learners  who  were  less  knowledgeable.  This  support  perhaps 
enabled  learners  to  better  tailor  their  learning  experience  to  their  identified  needs  without  having 
prior  knowledge  on  which  to  rely.  That  said,  another  possible  explanation  concerns  the  relative 
complexity  of  the  tasks.  Adjust  Indirect  Fire,  while  a  less  familiar  task,  was  also  more  sequential 
in  structure  than  Conduct  a  Defense  by  a  Squad.  The  particular  ways  in  which  the  design 
features  were  implemented  may  have  better  supported  learning  of  a  more  sequential  and  clearly 
structured  task.  Even  so,  additional  research  may  be  necessary  to  identify  the  precise 
mechanisms  associated  with  this  effect. 

User  Experiences  with  the  IMI 

When  Soldiers  completed  the  experiment,  they  filled  out  a  user  experience  questionnaire 
to  evaluate  the  needs-focused  IMI.  On  the  basis  of  those  responses,  an  exploratory  factor 
analysis  was  conducted  to  identify  the  critical  factors  supporting  their  learning  experience  with 
IMI.  There  were  no  significant  differences  between  the  familiarization,  core/refresher,  and 
tailored  IMI  variations  in  terms  of  Soldiers’  learning  experiences.  However,  the  targeted 
audience  tended  to  rate  their  learning  experience  with  the  IMI  more  positively.  The  items  and 
factors  identified  for  the  user  experience  questionnaire  may  be  useful  in  future  research  to 
evaluate  learner  experiences  and  to  prioritize  areas  for  improvement  in  existing  IMI.  The 
specific  items  and  factors  are  presented  in  Appendix  A. 
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IMI  Design  and  Development 

When  developing  IMI  for  this  research,  certain  design  principles  were  particularly  useful 
for  tailoring  content  to  individual  learners’  needs.  For  example,  a  hierarchical  and  chunked 
organizational  structure  enabled  users  to  quickly  address  a  single  topic  in  relation  to  other  topics 
covered  and  to  navigate  through  the  IMI.  Segmenting  and  presenting  content  in  this  way  may 
also  allow  learners  to  take  breaks  and  resume  learning  later  with  reduced  disruption  (Mayer, 
2005).  Soldiers  interviewed  said  that  it  is  important  that  IMI  be  designed  to  allow  learners  to 
take  breaks  from  the  IMI  and  later  resume  it,  given  the  many  competing  on-the-job  demands. 

Another  design  principle  of  the  IMI  applied  to  this  research  was  the  use  of  evaluation  and 
feedback  for  each  task  step,  similar  to  the  usual  checks-on-leaming  approach  used  by  the  Army, 
but  the  evaluation  and  feedback  focused  more  on  the  set  of  steps  to  execute  a  complete  task.  The 
intention  was  to  support  learners  in  developing  a  sequential  understanding  of  the  task,  providing 
rationale  for  each  step  and  how  those  steps  interrelate.  Further,  the  structure  of  the  task  steps  and 
decision  processes  at  each  point  was  a  central  focus.  This  was  easy  to  do  for  a  sequential  task 
like  Adjust  Indirect  Fire;  however,  it  was  more  difficult  when  considering  the  multiple 
simultaneous  subtasks  and  decision  points  in  Conduct  a  Defense  by  a  Squad.  The  Conduct  a 
Defense  by  a  Squad  IMI  modules  required  more  articulated  structures  in  order  to  capture  the 
various  narrative  threads  of  the  task  and  to  address  learners’  particular  needs.  So,  although  the 
Conduct  a  Defense  by  a  Squad  task  was  more  familiar  to  Soldiers,  it  also  required  the  IMI 
developed  to  be  more  complex  in  its  design  and  selection  of  content. 

Finally,  a  diagnostic  assessment  and  individualized  feedback  in  the  tailored  training 
condition  was  used  to  assist  learners  in  identifying  a  path  through  the  IMI  that  would  be 
optimized  for  their  learning  needs.  By  providing  detailed  feedback  to  learners  about  the  topic 
areas  in  which  they  had  difficulty  and  allowing  them  to  make  informed  decisions  about  how  they 
would  navigate  the  IMI,  it  was  possible  to  apply  tailored  training  principles  in  IMI  without 
relying  on  a  learning  management  system.  While  a  learning  management  system  is  often 
required  for  certain  types  of  adaptive  training,  the  approach  used  in  this  research  may  allow  for 
tailoring  within  even  less  complex  and  standalone  types  of  IMI. 

It  was  determined  that  only  30%  of  the  existing  Army  IMI  reviewed  was  reusable  in  the 
way  developers  had  intended  (see  Blankenbeckler  et  al.,  2014).  Therefore,  70%  of  the  IMI  used 
in  the  experiment  was  developed  from  a  cold  start.  This  suggests  that  the  Army  would  benefit 
from  a  continued  focus  on  IMI  research  and  the  need  to  reuse  IMI  across  a  variety  of  learning 
contexts  (i.e.,  Sharable  Content  Object  Reference  Model;  DoD,  2011).  There  are  many  excellent 
examples  of  Army  IMI  that  demonstrate  solid  instructional  design  and  application  of  learning 
science.  However,  few  are  currently  configured  for  or  are  available  in  a  format  that  facilitates 
modification  and  reuse  to  produce  learning  materials  for  different  courses  and  audiences.  When 
seeking  to  modify  and  reuse  existing  IMI,  this  research  encountered  a  number  of  challenges. 
Often,  source  files  were  difficult  to  track  down,  the  software  in  which  the  original  IMI  was 
designed  was  outdated,  and  uniforms,  doctrine,  etc.,  had  sometimes  changed.  In  order  to  reuse 
IMI,  workarounds  were  often  necessary.  As  an  example,  digitally  recording  audio  and  video  off 
the  My  Training  Tab  (MT2)  site  allowed  the  designers  to  produce  an  editable  file.  This  was  a 
more  effective  approach  than  attempting  to  modify  incomplete  or  outdated  source  files. 
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Unfortunately,  trying  to  modify  and  reuse  existing  IMI  was  more  time  consuming  than 
developing  IMI  from  a  cold  start. 

Researchers  have  demonstrated  that  designing  and  implementing  effective  IMI  requires 
designers  to  plan  how  to  present  content  in  advance.  During  this  planning  process,  consideration 
must  be  given  to  the  cognitive  capabilities  of  the  target  audience  (i.e.,  prior  knowledge,  attention, 
working  memory,  cognitive  load,  etc.),  as  well  as,  determining  which  design  principles  (i.e., 
segmentation,  scaffolding,  highlighting,  personalization,  etc.)  will  be  most  useful  given  these 
potential  individual  differences  within  the  target  audience. 

The  most  serious  problem  that  can  emerge  from  ineffective  multimedia  design  is 
cognitive  overload  (Mayer  &  Moreno,  2003).  Most  often,  cognitive  overload  results  from 
individuals  engaging  in  too  much  extraneous  processing  (Mayer,  2008).  Meaningful  learning 
requires  substantial  amounts  of  mental  effort  and  intentional  processing,  but  due  to  limited 
cognitive  resources  (e.g.,  working  memory  and/or  attention),  the  processing  demands  evoked  by 
multimedia  can  exceed  individuals’  processing  capabilities  (Mayer  &  Moreno,  2003).  To  be  a 
successful  instructional  tool,  IMI  and  multimedia  systems  must  be  adaptable  and  account  for 
differences  in  cognitive  capacities  among  learners  (Gyselinck  et  al.,  2000).  For  example, 
animations  can  reduce  extraneous  processing  and  increase  interactivity;  however,  some 
researchers  have  proposed  that  including  animations  in  multimedia  presentations  can  impede 
learning  by  creating  false  perceptions  about  the  ease  of  learning  from  multimedia  (i.e.,  illusions 
of  understanding;  Paik  &  Schraw,  2013).  If  individuals  think  something  is  easy  to  learn,  they 
may  not  invest  as  much  effort  into  learning. 

Conclusions 

Tailored  and  adaptive  training  techniques  have  been  applied  in  some  existing  Army  IMI. 
Often,  these  are  complex  techniques  that  automate  the  process  of  down-selecting  content  to  focus 
only  on  that  which  is  most  appropriate  to  an  individual  learner’s  background  knowledge  and 
experience.  While  these  approaches  can  be  very  effective,  they  frequently  require  a  large  data 
set,  or  are  costly  to  develop  (Blankenbeckler  et  al.,  2013).  When  the  audience  for  a  particular 
IMI  is  large,  it  is  reasonable  to  make  a  large  investment  in  its  development.  However,  for  a  more 
narrowly  defined  audience,  complex  implementations  of  tailored  or  adaptive  training  may  not  be 
viable  (Blankenbeckler  et  al.,  2014). 

Nonetheless,  streamlined  and  more  cost-effective  IMI  designs  may  be  applied  to  address 
the  learning  needs  of  narrowly  defined  audiences,  targeting  what  has  been  described  in  the  Army 
Learning  Model  (ALM)  as  point  of  need,  which  has  been  described  by  two  perspectives 
(TRADOC,  201 1).  The  first  perspective  focuses  on  the  technologies  used  to  provide  instruction 
where  and  when  needs  may  arise.  These  technologies  include  desktop  or  laptop  computers, 
tablets,  smartphones,  or  other  technologies  that  serve  to  distribute  or  access  digital  instructional 
materials.  The  second  perspective  on  point  of  need  focuses  on  the  content  and  design  of  the 
instruction  itself.  As  each  Soldier  comes  to  training  with  unique  professional  experiences, 
background  knowledge,  and  job- specific  requirements,  their  learning  needs  will  tend  to  differ. 
These  differences  can  affect  how  well  Soldiers  leam  with  regard  to  specific  educational  and 
training  contexts,  or  with  particular  subject  matter  and  media  (TRADOC,  201 1).  Research  is 
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needed  to  address  how  individual  differences  between  learners  can  best  be  accommodated  when 
designing  IMI  for  audiences  that  are  diverse  in  background  knowledge  and  experiences,  yet 
share  a  common  learning  need. 

Continued  research  and  innovation  is  essential  for  determining  how  to  best  develop  IMI 
that  can  be  modified  and  reused  across  different  learning  contexts  and  for  different  learning 
needs.  The  design  principles  identified  by  this  research  provide  a  simple  and  effective  means  to 
apply  tailored  training  techniques  within  IMI  that  do  not  require  complex  programming  and  high 
bandwidth  features.  This  streamlined  approach  to  needs-focused  IMI  design  may  be  applicable 
for  the  Army  and  other  branches  of  the  military.  When  learners  are  less  familiar  with  a  topic 
domain,  it  may  be  useful  to  provide  them  with  an  understandable  navigational  structure, 
including  clearly  labeled  chunks  of  content,  to  supplement  their  understanding  of  the  domain  and 
enable  them  to  select  an  appropriate  learning  path  in  order  to  tailor  their  learning  experience. 
Using  diagnostic  assessments  with  individualized  feedback  may  further  assist  learners  in 
discriminating  between  personally  relevant  and  irrelevant  content  while  learning,  even  if  they  do 
not  yet  have  extensive  domain  knowledge.  When  learners  are  more  familiar  with  a  topic,  the 
navigational  structure  appeared  to  assist  them  in  seeking  novel  information.  In  this  regard,  the 
information  seeking  behavior  observed  was  similar  to  what  has  been  observed  in  other  contexts 
when  comparing  novices  to  more  knowledgeable  learners  (Kalyuga,  2005). 
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Acronyms 


1SG 

First  Sergeant 

ALM 

Army  Learning  Model 

ANOVA 

Analysis  of  Variance 

ARI 

Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

CAT 

Computer  Adaptive  Testing 

COL 

Colonel 

CPT 

Captain 

DOTD 

Directorate  of  Training  and  Doctrine 

FDC 

Fire  Direction  Center 

FCoE 

Fires  Center  of  Excellence 

E-4 

Paygrade  for  Specialist/Corporal 

E-5 

Paygrade  for  Sergeant 

IMI 

Interactive  Multimedia  Instruction 

LTC 

Lieutenant  Colonel 

MCoE 

Maneuver  Center  of  Excellence 

MT2 

My  Training  Tab 

MOS 

Military  Occupational  Specialty 

NCO 

Noncommissioned  Officer 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

WLC 

Warrior  Leader  Course 
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APPENDIX  A 

USER  EXPERIENCE  QUESTIONNAIRE  FACTORS  AND  ITEMS 


A-l 


User  Experience  Questionnaire  Factors  and  Items 


User  Experience  Factors  User  Experience  Scores 


Mean 

(SD) 


Adjust  Indirect  Fire 

Conduct  a  Defense 
by  a  Squad 

Fam  Core  Tailored 

n= 14  h=14  h=16 

Fam  Core  Tailored 

n=  15  «=15  n=ll 

1.  Quality  of  Learning  Experience  ( a  =  0.91) 

•  I  would  recommend  that  this  IMI  be  made 
available  to  all  junior  NCOs. 

4.07 

(0.92) 

3.64 

(1.08) 

3.81 

(0.83) 

4.27 

(0.70) 

4.00 

(1.00) 

3.71 

(0.77) 

•  I  would  use  this  IMI  to  refresh  my  skills  at 
a  later  date. 

3.93 

(0.83) 

3.64 

(1.01) 

3.94 

(0.68) 

4.27 

(0.88) 

3.93 

(0.88) 

3.88 

(0.99) 

•  I  feel  I  have  a  better  understanding  of  the 
task  after  completing  the  IMI. 

3.86 

(1.03) 

3.57 

(0.94) 

4.00 

(0.82) 

4.07 

(0.96) 

4.27 

(0.59) 

3.71 

(0.77) 

•  I  preferred  this  IMI  to  others  I  have  used 
in  the  past. 

3.43 

(1.09) 

3.57 

(1.02) 

3.44 

(0.73) 

4.20 

(0.94) 

3.67 

(0.90) 

3.47 

(0.87) 

•  The  IMI  interactively  helped  my  learning 
process. 

3.86 

(1.03) 

3.71 

(0.73) 

3.88 

(0.62) 

3.80 

(0.86) 

4.07 

(0.59) 

3.76 

(0.83) 

•  On  the  basis  of  this  IMI,  I  could  execute 
the  task  as  a  combat  leader. 

3.64 

(1.01) 

3.21 

(1.19) 

3.63 

(1.20) 

3.60 

(1.35) 

3.73 

(0.96) 

3.35 

(1.22) 

•  I  feel  this  IMI  was  able  to  meet  my 
individual  learning  needs. 

3.71 

(0.83) 

3.50 

(0.94) 

3.88 

(0.96) 

4.00 

(0.76) 

3.80 

(0.86) 

3.35 

(0.93) 

Overall 

3.79 

(0.75) 

3.55 

(0.82) 

3.79 

(0.65) 

4.03 

(0.81) 

3.92 

(0.62) 

3.61 

(0.77) 

2.  Quality  of  Design  and  Content  (a  =  0.86) 

•  The  displays  on  the  screen  were  clear  and 
legible. 

4.50 

(0.52) 

4.29 

(0.61) 

4.31 

(0.60) 

4.53 

(0.64) 

4.33 

(0.62) 

4.06 

(0.75) 

•  The  graphics  supported  the  material  being 
presented. 

4.35 

(0.50) 

4.14 

(0.53) 

4.06 

(0.57) 

4.40 

(0.63) 

4.13 

(0.83) 

4.18 

(0.64) 

•  Prompts  and  cues  in  the  IMI  assisted  me 
in  navigating  through  the  material. 

4.07 

(0.83) 

4.00 

(0.68) 

3.88 

(0.62) 

4.13 

(0.92) 

4.27 

(0.46) 

4.06 

(0.66) 

•  The  information  presented  seemed 
accurate  and  doctrinally  correct. 

4.14 

(0.53) 

4.07 

(0.62) 

4.06 

(0.77) 

4.40 

(0.63) 

3.87 

(0.92) 

4.23 

(0.56) 

•  I  felt  like  I  was  in  control  of  my  learning 
process. 

4.21 

(0.80) 

3.57 

(1.02) 

3.88 

(0.72) 

4.27 

(0.88) 

4.00 

(0.65) 

3.65 

(0.93) 

•  The  information  presented  seemed  up-to- 
date. 

4.14 

(0.53) 

4.07 

(0.62) 

4.13 

(0.72) 

4.20 

(0.68) 

4.13 

(0.83) 

4.18 

(0.81) 

•  I  could  easily  track  where  I  was  in  the 

IMI. 

3.93 

(0.92) 

3.86 

(0.66) 

3.63 

(1.09) 

3.80 

(1.26) 

4.07 

(0.70) 

3.76 

(0.83) 

•  Uniforms,  practices,  and  equipment  were 
up  to  date. 

4.14 

(0.53) 

3.64 

(0.84) 

3.94 

(0.77) 

4.07 

(1.03) 

3.87 

(0.99) 

4.06 

(0.75) 

Overall 

4.19 

(0.46) 

3.96 

(0.54) 

3.98 

(0.56) 

4.23 

(0.65) 

4.08 

(0.50) 

4.02 

(0.50) 

3.  Continuity  of  Topics  (a  =  0.85) 

•  There  was  a  good  connection  between  the 
topics. 

4.29 

(0.61) 

3.79 

(0.70) 

4.00 

(0.52) 

4.33 

(0.72) 

4.20 

(0.56) 

3.88 

(0.70) 

•  The  sequence  of  topics  seemed  to  build  on 
each  other. 

4.21 

(0.58) 

3.71 

(0.83) 

3.88 

(0.72) 

4.20 

(0.86) 

4.13 

(0.64) 

3.71 

(0.69) 

A-2 


•  IMI  content  was  grouped  to  facilitate 
learning. 

4.14 

(0.66) 

4.07 

(0.62) 

4.06 

(0.57) 

4.33 

(0.49) 

4.20 

(0.68) 

4.06 

(0.66) 

•  There  was  a  clear  focus  of  topics  in  the 

IMI. 

4.43 

(0.65) 

4.07 

(0.62) 

4.13 

(0.62) 

4.53 

(0.52) 

4.13 

(0.74) 

4.00 

(0.79) 

•  Grouping  of  content  allowed  me 
flexibility  in  accessing  material. 

3.64 

(0.50) 

4.00 

(0.68) 

4.06 

(0.57) 

4.27 

(0.88) 

4.07 

(0.70) 

4.12 

(0.60) 

Overall 

4.14 

(0.47) 

3.93 

(0.57) 

4.03 

(0.48) 

4.33 

(0.57) 

4.15 

(0.55) 

3.95 

(0.52) 

4.  Credible  Examples  (a  =  0.85) 

•  Examples  contributed  to  my  learning. 

4.07 

(0.92) 

3.86 

(0.66) 

4.13 

(0.62) 

4.20 

(0.68) 

4.27 

(0.59) 

4.06 

(0.83) 

•  The  examples  made  sense. 

4.21 

(0.43) 

4.00 

(0.55) 

4.19 

(0.54) 

4.20 

(0.68) 

4.20 

(0.56) 

4.06 

(0.66) 

•  I  learned  a  lot  about  the  task. 

3.71 

(0.91) 

3.50 

(0.76) 

3.88 

(0.81) 

4.00 

(0.93) 

4.07 

(0.80) 

3.82 

(0.64) 

•  Examples  were  presented  in  a  realistic 
mission  context. 

4.21 

(0.58) 

4.00 

(0.68) 

4.25 

(0.58) 

4.00 

(0.76) 

3.93 

(0.80) 

4.06 

(0.56) 

•  Repetition  of  examples  was  helpful. 

4.07 

(0.62) 

4.00 

(0.68) 

3.94 

(0.68) 

4.20 

(0.68) 

4.07 

(0.59) 

4.12 

(0.78) 

•  Topics  were  of  the  right  length  to  allow 
me  to  complete  without  needing  a  break. 

3.93 

(0.62) 

3.93 

(0.73) 

4.06 

(0.68) 

4.33 

(0.49) 

4.07 

(0.80) 

3.59 

(0.80) 

Overall 

4.04 

(0.54) 

3.88 

(0.54) 

4.07 

(0.58) 

4.16 

(0.53) 

4.10 

(0.49) 

3.95 

(0.47) 

5.  Focus  and  Relevance  (a  =  0.77) 

•  Questions  asked  within  the  IMI  were 
reasonable  and  helped  me  to  understand 
the  topic. 

4.07 

(0.73) 

3.71 

(0.91) 

3.81 

(0.75) 

4.20 

(0.76) 

4.00 

(0.76) 

4.00 

(0.71) 

•  The  questions  asked  within  the  IMI 
focused  on  what  was  being  taught. 

3.93 

(0.62) 

3.93 

(0.83) 

3.94 

(0.75) 

4.33 

(0.62) 

4.20 

(0.68) 

4.12 

(0.49) 

•  The  overall  focus  of  the  IMI  was  right  on 
target. 

4.21 

(0.58) 

3.86 

(0.86) 

4.06 

(0.77) 

4.27 

(0.59) 

4.00 

(0.85) 

4.24 

(0.66) 

Overall 

4.07 

(0.49) 

3.83 

(0.74) 

3.94 

(0.73) 

4.27 

(0.54) 

4.07 

(0.58) 

4.12 

(0.44) 

6.  Tracking  Progress  (a  =  0.77) 

•  If  I  took  a  break  during  the  learning 
process,  I  could  easily  resume  learning 
when  I  returned. 

4.14 

(0.77) 

3.79 

(0.80) 

3.75 

(0.93) 

3.93 

(0.96) 

3.93 

(1.10) 

3.65 

(0.86) 

•  I  would  be  able  to  take  breaks  during  the 
learning  process  and  keep  track  of  my 
progress. 

3.93 

(0.83) 

3.50 

(0.96) 

3.56 

(0.96) 

3.60 

(0.91) 

3.73 

(0.80) 

3.65 

(0.79) 

Overall 

4.03 

(0.75) 

3.64 

(0.77) 

3.66 

(0.91) 

3.77 

(0.82) 

3.83 

(0.88) 

3.65 

(0.77) 
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